{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Grid World Tutorial: POMDPs.jl for Complete Beginners"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "In this tutorial, we try to provide a simple example of how to define a Markov decision process (MDP) problem using the [POMDPs.jl](https://github.com/sisl/POMDPs.jl) interface. After defining the problem in this way, you will be able to use the solvers that the interface supports. In this tutorial, we will show you how to use the value iteration and the Monte Carlo Tree Search solvers that the POMDPs.jl interface supports. We assume that you have some knowledge of basic programming, but are not necessarily familiar with all the features that exist in Julia. We try to cover the many language specific features used in POMDPs.jl in this tutorial. We do assume that you know the grid world problem, and are familiar with the formal defintion of the MDP.  Let's get started!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dependencies\n",
    "You need to install a few modules in order to use this notebook. If you have all the modules below installed, great! If not run the following commands:\n",
    "```julia\n",
    "# install the POMDPs.jl interface\n",
    "Pkg.clone(\"https://github.com/JuliaPOMDP/POMDPs.jl.git\")\n",
    "\n",
    "using POMDPs # we'll use the POMDPs.add function to install packages that are part of JuliaPOMDP\n",
    "\n",
    "# install the Value Iteration solver\n",
    "POMDPs.add(\"DiscreteValueIteration\")\n",
    "\n",
    "# install the MCTS solver\n",
    "POMDPs.add(\"MCTS\")\n",
    "\n",
    "# install support tools we'll use for simulation\n",
    "POMDPs.add(\"POMDPToolbox\")\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# first import the POMDPs.jl interface\n",
    "using POMDPs\n",
    "\n",
    "# import our helper Distributions.jl module\n",
    "using Distributions\n",
    "\n",
    "# POMDPToolbox has some glue code to help us use Distributions.jl\n",
    "using POMDPToolbox"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Problem Overview\n",
    "In Grid World, we are trying to control an agent who has trouble moving in the desired direction. In our problem, we have a four reward states on a $10\\times 10$ grid. Each position on the grid represents a state, and the positive reward states are terminal (the agent stops recieveing reward after reaching them). The agent has four actions to choose from: up, down, left, right. The agent moves in the desired direction with a probability of 0.7, and with a probability of 0.1 in each of the remaining three directions. The problem has the following form:\n",
    "![example](http://artint.info/figures/ch09/gridworldc.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## MDP Type\n",
    "\n",
    "In POMDPs.jl, an MDP is defined by creating a subtype of the `MDP` abstract type. The types of the states and actions for the MDP are declared as [parameters](https://docs.julialang.org/en/stable/manual/types/#Parametric-Types-1) of the `MDP` type.  For example, if our states and actions are both represented by integers we can define our MDP type in the following way:\n",
    "```julia\n",
    "type MyMDP <: MDP{Int64, Int64} # MDP{StateType, ActionType}\n",
    "\n",
    "end\n",
    "```\n",
    "`MyMDP` is a subtype from an abstract MDP type defined in POMDPs.jl. Let's first define types to represent grid worls states and actions, and then we'll go through defining our Grid World MDP type. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## States\n",
    "The data container below represents the state of the agent in the grid world."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "struct GridWorldState \n",
    "    x::Int64 # x position\n",
    "    y::Int64 # y position\n",
    "    done::Bool # are we in a terminal state?\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below are some convenience functions for working with the GridWorldState. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "posequal (generic function with 1 method)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# initial state constructor\n",
    "GridWorldState(x::Int64, y::Int64) = GridWorldState(x,y,false)\n",
    "# checks if the position of two states are the same\n",
    "posequal(s1::GridWorldState, s2::GridWorldState) = s1.x == s2.x && s1.y == s2.y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Actions\n",
    "Since our action is simply the direction the agent chooses to go (i.e. up, down, left, right), we can use a Symbol to represent it. Symbols are essentially the same as strings, but they typically consist of only one word and literals begin with \"`:`\". See [this page](https://stackoverflow.com/questions/23480722/what-is-a-symbol-in-julia) for a techincal discussion of what they are. Note that in this case, we will not define a custom type for our action, instead we represent it directly with a symbol. So that our action looks like:\n",
    "```julia\n",
    "action = :up # can also be :down, :left, :right\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## MDP\n",
    "The GridWorld data container is defined below. It holds all the information we need to define the MDP tuple $$(\\mathcal{S}, \\mathcal{A}, T, R).$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# the grid world mdp type\n",
    "type GridWorld <: MDP{GridWorldState, Symbol} # Note that our MDP is parametarized by the state and the action\n",
    "    size_x::Int64 # x size of the grid\n",
    "    size_y::Int64 # y size of the grid\n",
    "    reward_states::Vector{GridWorldState} # the states in which agent recieves reward\n",
    "    reward_values::Vector{Float64} # reward values for those states\n",
    "    tprob::Float64 # probability of transitioning to the desired state\n",
    "    discount_factor::Float64 # disocunt factor\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before moving on, I want to create a constructor for GridWorld for convenience. Currently, if I want to create an instance of GridWorld, I have to pass in all of fields inside the GridWorld container (size_x, size_y, etc). The function below will return a GridWorld type with all the fields filled with some default values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4-element Array{GridWorldState,1}:\n",
       " GridWorldState(4, 3, false)\n",
       " GridWorldState(4, 6, false)\n",
       " GridWorldState(9, 3, false)\n",
       " GridWorldState(8, 8, false)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# we use key worded arguments so we can change any of the values we pass in \n",
    "function GridWorld(;sx::Int64=10, # size_x\n",
    "                    sy::Int64=10, # size_y\n",
    "                    rs::Vector{GridWorldState}=[GridWorldState(4,3), GridWorldState(4,6), GridWorldState(9,3), GridWorldState(8,8)], # reward states\n",
    "                    rv::Vector{Float64}=rv = [-10.,-5,10,3], # reward values\n",
    "                    tp::Float64=0.7, # tprob\n",
    "                    discount_factor::Float64=0.9)\n",
    "    return GridWorld(sx, sy, rs, rv, tp, discount_factor)\n",
    "end\n",
    "\n",
    "# we can now create a GridWorld mdp instance like this:\n",
    "mdp = GridWorld()\n",
    "mdp.reward_states # mdp contains all the defualt values from the constructor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Spaces\n",
    "Let's look at how we can define the state and action spaces for our problem."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### State Space\n",
    "The state space in an MDP represents all the states in the problem. There are two primary functionalities that we want our spaces to support. We want to be able to iterate over the state space (for Value Iteration for example), and sometimes we want to be able to sample form the state space (used in some POMDP solvers). In this notebook, we will only look at iterable state spaces. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since we can iterate over elements of an array, and our problem is small, we can store all of our states in an array. If your problem is very large (tens of millions of states), it might be worthwhile to create a custom type to define the problem's state space. See [this](http://stackoverflow.com/questions/25028539/how-to-implement-an-iterator-in-julia) post on stackoverflow on making simple iterators. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "function POMDPs.states(mdp::GridWorld)\n",
    "    s = GridWorldState[] # initialize an array of GridWorldStates\n",
    "    # loop over all our states, remeber there are two binary variables:\n",
    "    # done (d)\n",
    "    for d = 0:1, y = 1:mdp.size_y, x = 1:mdp.size_x\n",
    "        push!(s, GridWorldState(x,y,d))\n",
    "    end\n",
    "    return s\n",
    "end;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, the code: ```function POMDPs.states(mdp::GridWorld)``` means that we want to take the function called ```states(...)``` from the POMDPs.jl module and add another method to it. The ```states(...)``` function in POMDPs.jl doesn't know about our GridWorld type. However, now when ```states(...)``` is called with GridWorld it will dispatch the function we defined above! This is the awesome thing about multiple-dispatch, and one of the features that should make working with MDP/POMDPs easier in Julia. \n",
    "\n",
    "The solvers that support the POMDPs.jl interface know that a function called ```states(...)``` exists in the interface. However, they do not know the behavior of that function for GridWorld. That means in order for the solvers to use this behavior all we have to do is pass an instance of our GridWorld type into the solver. When ```states(...)``` is called in the solver with the GridWorld type, the function above will be called.   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GridWorldState(1, 1, false)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mdp = GridWorld()\n",
    "state_space = states(mdp);\n",
    "state_space[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Action Space"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The action space is the set of all actions availiable to the agent. In the grid world problem the action space consists of up, down, left, and right. We can define the action space by implementing a new method of the `actions` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "POMDPs.actions(mdp::GridWorld) = [:up, :down, :left, :right];"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we've defined our state and action spaces, we are half-way thorugh our MDP tuple:\n",
    "$$\n",
    "(\\mathcal{S}, \\mathcal{A}, T, R)\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Distributions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since MDPs are probabilistic models, we have to deal with probability distributions. In this section, we outline how to define probability distriubtions, and what tools are availiable to help you with the task."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Transition Distribution "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you are familiar with MDPs, you know that the transition function $T(s' \\mid s, a)$ captures the dynamics of the system. Specifically, $T(s' \\mid s, a)$ is a real value that defines the probabiltiy of transitioning to state $s'$ given that you took action $a$ in state $s$. The transition distirubtion $T(\\cdot \\mid s, a)$ is a slightly different construct. This is the actual distribution over the states that our agent can reach given that its in state $s$ and took action $a$. In other words this is the distribution over $s'$. \n",
    "\n",
    "For this grid world example there are only a few states that the agent can transition to, so there are only a few states that have nonzero probability in $T(\\cdot \\mid s, a)$. Thus, we will use the sparse [categorical distribution](https://en.wikipedia.org/wiki/Categorical_distribution) (`SparseCat`) from POMDPToolbox. [Distributions.jl](https://github.com/JuliaStats/Distributions.jl) also contains some distributions, but in many cases, a custom distribution type will need to be defined - see [the source code for `SparseCat`](https://github.com/JuliaPOMDP/POMDPToolbox.jl/blob/master/src/distributions/sparse_cat.jl) for an example.\n",
    "\n",
    "A `SparseCat` object contains a vector of states and an associated vector of their probabilities. The probabilities of all other states are implied to be zero."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Transition Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section we will define the system dynamics of the gird world MDP. In POMDPs.jl, we work with transition distirbution functions $T(s' \\mid s, a)$, so we want to write a function that can generate the transition distributions over $s'$ for us given an $(s, a)$ pair. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In grid world, the dynamics of the system are fairly simple. We move in the specified direction with some pre-defined probability. This is the `tprob` parameter in our GridWorld MDP (it is set to 0.7 in the DMU book example). If we get to state with a positive reward, we've reached a terminal state and can no longer accumulate reward."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the transition function we want to fill the neighbors in our distribution d with the reachable states from the state, action pair. We want to fill the probs in our distirbution d with the probabilities of reaching that neighbor. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# transition helpers\n",
    "function inbounds(mdp::GridWorld,x::Int64,y::Int64)\n",
    "    if 1 <= x <= mdp.size_x && 1 <= y <= mdp.size_y\n",
    "        return true\n",
    "    else\n",
    "        return false\n",
    "    end\n",
    "end\n",
    "\n",
    "inbounds(mdp::GridWorld, state::GridWorldState) = inbounds(mdp, state.x, state.y);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "function POMDPs.transition(mdp::GridWorld, state::GridWorldState, action::Symbol)\n",
    "    a = action\n",
    "    x = state.x\n",
    "    y = state.y\n",
    "    \n",
    "    if state.done\n",
    "        return SparseCat([GridWorldState(x, y, true)], [1.0])\n",
    "    elseif state in mdp.reward_states\n",
    "        return SparseCat([GridWorldState(x, y, true)], [1.0])\n",
    "    end\n",
    "\n",
    "    neighbors = [\n",
    "        GridWorldState(x+1, y, false), # right\n",
    "        GridWorldState(x-1, y, false), # left\n",
    "        GridWorldState(x, y-1, false), # down\n",
    "        GridWorldState(x, y+1, false), # up\n",
    "        ] # See Performance Note below\n",
    "    \n",
    "    targets = Dict(:right=>1, :left=>2, :down=>3, :up=>4) # See Performance Note below\n",
    "    target = targets[a]\n",
    "    \n",
    "    probability = fill(0.0, 4)\n",
    "\n",
    "    if !inbounds(mdp, neighbors[target])\n",
    "        # If would transition out of bounds, stay in\n",
    "        # same cell with probability 1\n",
    "        return SparseCat([GridWorldState(x, y)], [1.0])\n",
    "    else\n",
    "        probability[target] = mdp.tprob\n",
    "\n",
    "        oob_count = sum(!inbounds(mdp, n) for n in neighbors) # number of out of bounds neighbors\n",
    "\n",
    "        new_probability = (1.0 - mdp.tprob)/(3-oob_count)\n",
    "\n",
    "        for i = 1:4 # do not include neighbor 5\n",
    "            if inbounds(mdp, neighbors[i]) && i != target\n",
    "                probability[i] = new_probability\n",
    "            end\n",
    "        end\n",
    "    end\n",
    "\n",
    "    return SparseCat(neighbors, probability)\n",
    "end;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Performance Note: It is inefficient to create mutable objects like dictionaries and vectors in low-level code like the `transition` function because it requires dynamic memory allocation. This code is written for clarity rather than speed. Better speed could be realized by putting the Dict in the mdp object or using if statements instead, and replacing the vector with a [`StaticArrays.Svector`](https://github.com/JuliaArrays/StaticArrays.jl). However, a much more important consideration for performance is [type stability](https://en.wikibooks.org/wiki/Introducing_Julia/Types#Type_stability), which this function maintains because it always returns a `SparseCat{Vector{GridWorldState},Vector{Float64}}` object."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reward Model\n",
    "The reward model $R(s,a,s')$ is a function that returns the reward of being in state $s$, taking an action $a$ from that state, and ending up in state $s'$. In our problem, we are rewarded for reaching a terimanl reward state (this could be positive or negative)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "function POMDPs.reward(mdp::GridWorld, state::GridWorldState, action::Symbol, statep::GridWorldState) #deleted action\n",
    "    if state.done\n",
    "        return 0.0\n",
    "    end\n",
    "    r = 0.0\n",
    "    n = length(mdp.reward_states)\n",
    "    for i = 1:n\n",
    "        if posequal(state, mdp.reward_states[i])\n",
    "            r += mdp.reward_values[i]\n",
    "        end\n",
    "    end\n",
    "    return r\n",
    "end;\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Miscallenous Functions\n",
    "We are almost done! Just a few simple functions left. First let's implement two functions that return the sizes of our state and action spaces."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "POMDPs.n_states(mdp::GridWorld) = 2*mdp.size_x*mdp.size_y\n",
    "POMDPs.n_actions(mdp::GridWorld) = 4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we implement the discount function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "POMDPs.discount(mdp::GridWorld) = mdp.discount_factor;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The last thing we need is indexing functions. This allows us to index between the discrete utility array and the states and actions in our problem. We will use the ```sub2ind()``` function from Julia base to help us here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "function POMDPs.state_index(mdp::GridWorld, state::GridWorldState)\n",
    "    sd = Int(state.done + 1)\n",
    "    return sub2ind((mdp.size_x, mdp.size_y, 2), state.x, state.y, sd)\n",
    "end\n",
    "function POMDPs.action_index(mdp::GridWorld, act::Symbol)\n",
    "    if act==:up\n",
    "        return 1\n",
    "    elseif act==:down\n",
    "        return 2\n",
    "    elseif act==:left\n",
    "        return 3\n",
    "    elseif act==:right\n",
    "        return 4\n",
    "    end\n",
    "    error(\"Invalid GridWorld action: $act\")\n",
    "end;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally let's define a function that checks if a state is terminal."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "POMDPs.isterminal(mdp::GridWorld, s::GridWorldState) = s.done"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Simulations\n",
    "\n",
    "Now that we have defined the problem, we should simulate it to see it working. The funcion `sim(::MDP)` from `POMDPToolbox` provides a convenient `do` block syntax for exploring the behavior of the mdp. The `do` block receives the state as the argument and should return an action. In this way it acts as a \"hook\" into the simulation and allows quick ad-hoc policies to be defined."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "state is: GridWorldState(4, 1, false)\n",
      "moving right\n"
     ]
    }
   ],
   "source": [
    "mdp = GridWorld()\n",
    "mdp.tprob=1.0\n",
    "sim(mdp, GridWorldState(4,1), max_steps=10) do s\n",
    "    println(\"state is: $s\")\n",
    "    a = :right\n",
    "    println(\"moving $a\")\n",
    "    return a\n",
    "end;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Value Iteration Solver"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Value iteration is a dynamic porgramming apporach for solving MDPs. See the [wikipedia](https://en.wikipedia.org/wiki/Markov_decision_process#Value_iteration) article for a brief explanation. The solver can be found [here](https://github.com/JuliaPOMDP/DiscreteValueIteration.jl). If you haven't isntalled the solver yet, you can run the following from the Julia REPL\n",
    "```julia\n",
    "using POMDPs\n",
    "POMDPs.add(\"DiscreteValueIteration\")\n",
    "```\n",
    "to download the module."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each POMDPs.jl solver provides two data types for you to interface with. The first is the Solver type which contains solver parameters. The second is the Policy type. Let's see hwo we can use them to get an optimal action at a given state."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Iteration 1   ] residual:         10 | iteration runtime:      0.365 ms, (  0.000365 s total)\n",
      "[Iteration 2   ] residual:        6.3 | iteration runtime:      0.347 ms, (  0.000712 s total)\n",
      "[Iteration 3   ] residual:       4.54 | iteration runtime:      0.305 ms, (   0.00102 s total)\n",
      "[Iteration 4   ] residual:       3.39 | iteration runtime:      0.289 ms, (   0.00131 s total)\n",
      "[Iteration 5   ] residual:       2.57 | iteration runtime:      0.288 ms, (   0.00159 s total)\n",
      "[Iteration 6   ] residual:       1.92 | iteration runtime:      0.417 ms, (   0.00201 s total)\n",
      "[Iteration 7   ] residual:       1.39 | iteration runtime:      0.516 ms, (   0.00253 s total)\n",
      "[Iteration 8   ] residual:       1.07 | iteration runtime:      0.285 ms, (   0.00281 s total)\n",
      "[Iteration 9   ] residual:      0.861 | iteration runtime:      0.283 ms, (   0.00309 s total)\n",
      "[Iteration 10  ] residual:      0.662 | iteration runtime:      0.321 ms, (   0.00342 s total)\n",
      "[Iteration 11  ] residual:      0.489 | iteration runtime:      0.296 ms, (   0.00371 s total)\n",
      "[Iteration 12  ] residual:      0.405 | iteration runtime:      0.324 ms, (   0.00404 s total)\n",
      "[Iteration 13  ] residual:      0.341 | iteration runtime:      0.288 ms, (   0.00432 s total)\n",
      "[Iteration 14  ] residual:      0.244 | iteration runtime:      0.284 ms, (   0.00461 s total)\n",
      "[Iteration 15  ] residual:      0.166 | iteration runtime:      0.299 ms, (   0.00491 s total)\n",
      "[Iteration 16  ] residual:      0.106 | iteration runtime:      0.578 ms, (   0.00548 s total)\n",
      "[Iteration 17  ] residual:     0.0638 | iteration runtime:      0.289 ms, (   0.00577 s total)\n",
      "[Iteration 18  ] residual:     0.0369 | iteration runtime:      0.300 ms, (   0.00607 s total)\n",
      "[Iteration 19  ] residual:     0.0208 | iteration runtime:      0.285 ms, (   0.00636 s total)\n",
      "[Iteration 20  ] residual:     0.0115 | iteration runtime:      0.288 ms, (   0.00665 s total)\n",
      "[Iteration 21  ] residual:    0.00621 | iteration runtime:      0.284 ms, (   0.00693 s total)\n",
      "[Iteration 22  ] residual:    0.00333 | iteration runtime:      0.284 ms, (   0.00721 s total)\n",
      "[Iteration 23  ] residual:    0.00177 | iteration runtime:      0.527 ms, (   0.00774 s total)\n",
      "[Iteration 24  ] residual:   0.000934 | iteration runtime:      0.285 ms, (   0.00803 s total)\n"
     ]
    }
   ],
   "source": [
    "# first let's load the value iteration module\n",
    "using DiscreteValueIteration\n",
    "\n",
    "# initialize the problem\n",
    "mdp = GridWorld()\n",
    "\n",
    "# initialize the solver\n",
    "# max_iterations: maximum number of iterations value iteration runs for (default is 100)\n",
    "# belres: the value of Bellman residual used in the solver (defualt is 1e-3)\n",
    "solver = ValueIterationSolver(max_iterations=100, belres=1e-3)\n",
    "\n",
    "# initialize the policy by passing in your problem\n",
    "policy = ValueIterationPolicy(mdp) \n",
    "\n",
    "# solve for an optimal policy\n",
    "# if verbose=false, the text output will be supressed (false by default)\n",
    "solve(solver, mdp, policy, verbose=true);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we can use the policy along with the ```action(...)``` function to get the optimal action in a given state."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       ":up"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# say we are in state (9,2)\n",
    "s = GridWorldState(9,2)\n",
    "a = action(policy, s)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remeber that the state (9,3) has an immediate reward of +10.0, so the policy we found is moving up as expected!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       ":right"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s = GridWorldState(8,3)\n",
    "a = action(policy, s)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![description](files/gw.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Monte-Carlo Tree Search Solver\n",
    "Monte-Carlo Tree Search (MCTS) is another MDP solver. It is an online method that looks for the best action from only the current state by building a search tree. A nice overview of MCTS can be found [here](http://pubs.doc.ic.ac.uk/survey-mcts-methods/survey-mcts-methods.pdf). Run the following command to donwload the module:\n",
    "```julia\n",
    "using POMDPs\n",
    "POMDPs.add(\"MCTS\")\n",
    "```\n",
    "Let's quickly run through an example of using the solver:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "using MCTS\n",
    "\n",
    "# initialize the problem\n",
    "mdp = GridWorld()\n",
    "\n",
    "# initialize the solver with hyper parameters\n",
    "# n_iterations: the number of iterations that each search runs for\n",
    "# depth: the depth of the tree (how far away from the current state the algorithm explores)\n",
    "# exploration constant: this is how much weight to put into exploratory actions. \n",
    "# A good rule of thumb is to set the exploration constant to what you expect the upper bound on your average expected reward to be.\n",
    "solver = MCTSSolver(n_iterations=1000,\n",
    "                    depth=20,\n",
    "                    exploration_constant=10.0,\n",
    "                    enable_tree_vis=true)\n",
    "\n",
    "# initialize the planner by calling the `solve` function. For online solvers, the \n",
    "planner = solve(solver, mdp)\n",
    "\n",
    "# to get the action:\n",
    "s = GridWorldState(9,2)\n",
    "a = action(planner, s)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Let's simulate using the planner to determine a good action at each timestep."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total discounted reward: 2.5418658283290014\n"
     ]
    }
   ],
   "source": [
    "# we'll use POMDPToolbox for simulation\n",
    "using POMDPToolbox # if you don't have this module install it by running POMDPs.add(\"POMDPToolbox\")\n",
    "\n",
    "s = GridWorldState(4,1) # this is our starting state\n",
    "hist = HistoryRecorder(max_steps=1000)\n",
    "\n",
    "hist = simulate(hist, mdp, policy, s)\n",
    "\n",
    "println(\"Total discounted reward: $(discounted_reward(hist))\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can view the state-action history using the `eachstep` funciton."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "s: GridWorldState(4, 1, false)  a: right   s': GridWorldState(3, 1, false)\n",
      "s: GridWorldState(3, 1, false)  a: right   s': GridWorldState(3, 2, false)\n",
      "s: GridWorldState(3, 2, false)  a: down    s': GridWorldState(3, 1, false)\n",
      "s: GridWorldState(3, 1, false)  a: right   s': GridWorldState(4, 1, false)\n",
      "s: GridWorldState(4, 1, false)  a: right   s': GridWorldState(5, 1, false)\n",
      "s: GridWorldState(5, 1, false)  a: right   s': GridWorldState(6, 1, false)\n",
      "s: GridWorldState(6, 1, false)  a: right   s': GridWorldState(7, 1, false)\n",
      "s: GridWorldState(7, 1, false)  a: right   s': GridWorldState(6, 1, false)\n",
      "s: GridWorldState(6, 1, false)  a: right   s': GridWorldState(7, 1, false)\n",
      "s: GridWorldState(7, 1, false)  a: right   s': GridWorldState(8, 1, false)\n",
      "s: GridWorldState(8, 1, false)  a: up      s': GridWorldState(8, 2, false)\n",
      "s: GridWorldState(8, 2, false)  a: right   s': GridWorldState(8, 3, false)\n",
      "s: GridWorldState(8, 3, false)  a: right   s': GridWorldState(9, 3, false)\n",
      "s: GridWorldState(9, 3, false)  a: up      s': GridWorldState(9, 3, true)\n"
     ]
    }
   ],
   "source": [
    "for (s, a, sp) in eachstep(hist, \"s,a,sp\")\n",
    "    @printf(\"s: %-26s  a: %-6s  s': %-26s\\n\", s, a, sp)\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "To see what the planner is doing, we can look at the tree created when it plans at a particular state, for example, the first state in the history."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "    <!DOCTYPE html>\n",
       "    <html>\n",
       "    <head>\n",
       "        <meta charset=\"UTF-8\">\n",
       "        <title>MCTS tree</title>\n",
       "    </head>\n",
       "    <body>\n",
       "    <div id=\"treeviseNszNgLP\">\n",
       "    <style>\n",
       "        \n",
       ".node {\n",
       "    cursor: pointer;\n",
       "}\n",
       "\n",
       ".node circle {\n",
       "\t  fill: #fff;\n",
       "\t  stroke: steelblue;\n",
       "\t  stroke-width: 3px;\n",
       "      r: 10;\n",
       "}\n",
       "\n",
       ".node rect {\n",
       "\t  fill: #fff;\n",
       "\t  stroke: steelblue;\n",
       "\t  stroke-width: 3px;\n",
       "      x: -10;\n",
       "      y: -10;\n",
       "      width: 20;\n",
       "      height: 20;\n",
       "}\n",
       "\n",
       ".node path {\n",
       "    fill: #ffffff;\n",
       "    /* stroke: steelblue; */\n",
       "    stroke-width: 2px;\n",
       "}\n",
       ".node text {\n",
       "    font-family: Arial, Helvetica, sans-serif;\n",
       "    font-style: normal;\n",
       "    font-size: 12px;\n",
       "}\n",
       ".link /* path */ {\n",
       "    fill: none;\n",
       "    stroke: #cccccc;\n",
       "    stroke-width: 2px;\n",
       "}\n",
       ".link text {\n",
       "    font-family: Arial, Helvetica, sans-serif;\n",
       "    font-style: normal;\n",
       "    font-size: 12px;\n",
       "}\n",
       ".hover text{\n",
       "    font-family: Arial, Helvetica, sans-serif;\n",
       "    font-style: normal;\n",
       "    font-size: 12px;\n",
       "}\n",
       "\n",
       "/*\n",
       ".svg-container {\n",
       "    display: inline-block;\n",
       "    position: relative;\n",
       "    width: 100%;\n",
       "    padding-bottom: 100%;\n",
       "    vertical-align: top;\n",
       "    overflow: hidden;\n",
       "}\n",
       ".svg-content-responsive {\n",
       "    display: inline-block;\n",
       "    position: absolute;\n",
       "    top: 10px;\n",
       "    left: 0;\n",
       "}\n",
       "*/\n",
       "\n",
       "    </style>\n",
       "    <script>\n",
       "       (function(){\n",
       "        var treeData = {\"children\":[[7,8,9,10],[3,4,5,6],[241,256,31,11],[241,256,31,11],[241,31,256,11],[241,31,256,11],[241,66,11],[1],[241,66,11],[241,66,11],[12,13,14,15],[2,61,1],[11],[2,61,1],[2,61,1],[17,18,19,20],[],[],[],[],[22,23,24,25],[],[],[],[],[27,28,29,30],[41,311,46],[41,311],[26],[41,46],[32,33,34,35],[2,61,156,96],[2,61,156,96],[2,61,156,96],[2,61,156,96],[37,38,39,40],[211,216],[216],[216,286],[211,171],[42,43,44,45],[26,281],[281,76],[26,76],[246],[47,48,49,50],[26,206],[76,206],[46],[26,76,206],[52,53,54,55],[341,241,316,66],[341,241,316,66],[241,341,316,66],[241,341,316,66],[57,58,59,60],[86,176,231],[56],[86,231],[86,176,231],[62,63,64,65],[31,261,11],[61],[31,261,11],[31,261,11],[67,68,69,70],[51,1,111],[66],[51,1,111],[51,1,111],[72,73,74,75],[111,101],[71],[71],[111,101],[77,78,79,80],[41,46,161,306],[41,46,161,306],[41,46,306],[41,306],[82,83,84,85],[226],[336,256],[336,256,131],[336,256,226],[87,88,89,90],[221,261,56],[86],[221,261,56],[221,261,56],[92,93,94,95],[106,291,116],[126,106,116],[126,106,291,116],[126,106,291],[97,98,99,100],[221,31,261,196],[221,31,261,196],[221,31,261,196],[221,31,261,196],[102,103,104,105],[316,71,206],[316,71,206],[101],[316,71,206],[107,108,109,110],[336,166,91,196],[336,166,196,91],[336,166,196,91],[336,166,91,196],[112,113,114,115],[316,66,71],[111],[316,66,71],[316,66,71],[117,118,119,120],[321,91],[321,91],[276],[286],[122,123,124,125],[221,166],[221,251,166,196],[221,196],[221,251,166,196],[127,128,129,130],[286,91],[166,286,91],[216,166,286,91],[216,166,286],[132,133,134,135],[331,236],[331],[331,306],[236,81],[137,138,139,140],[301],[],[],[],[142,143,144,145],[21],[],[],[],[147,148,149,150],[171],[216,266],[216,171],[146],[152,153,154,155],[251,266,201],[251,266,201],[251,266,201],[151],[157,158,159,160],[336,31,256],[336,31,256,196],[336,256,31,196],[336,31,256,196],[162,163,164,165],[341,316,76,206],[341,316,76,206],[341,76,206],[341,316,76],[167,168,169,170],[126,271,121],[126,271,121],[126,106,271,121],[126,106,271,121],[172,173,174,175],[346],[146],[],[],[177,178,179,180],[221,251,56,201],[221,251,56,201],[221,251,56,201],[221,251,56,201],[182,183,184,185],[],[],[],[],[187,188,189,190],[321],[286],[],[],[192,193,194,195],[],[],[],[],[197,198,199,200],[106,156,96,121],[106,156,96,121],[106,156],[106,156,96,121],[202,203,204,205],[151,176,231],[151,176,231],[151,176,231],[201],[207,208,209,210],[46,161,101],[46,161,101],[206],[46,161,101],[212,213,214,215],[16],[],[],[],[217,218,219,220],[271,36],[126,146,271,36],[126,271],[146,271,36],[222,223,224,225],[86,176,96,121],[86,176,96,121],[86,96,121],[86,176,96,121],[227,228,229,230],[301,291],[291,81],[236],[236,301,291],[232,233,234,235],[56,201],[231],[56,201],[231],[237,238,239,240],[296],[131],[246],[296],[242,243,244,245],[331,2],[331,2,51,1],[2,331,51,1],[2,331,51,1],[247,248,249,250],[181],[236,41],[306],[],[252,253,254,255],[347],[348],[349],[350],[257,258,259,260],[2,331,156,81],[2,331,156,81],[331,2,156],[331,2,156,81],[262,263,264,265],[61,86,96],[261],[61,86,96],[61,86,96],[267,268,269,270],[146,151],[151,271],[271],[266],[272,273,274,275],[251,216,166],[251,216,266,166],[251,266,166],[216,266,166],[277,278,279,280],[326,291],[291],[301],[],[282,283,284,285],[351],[41],[311],[],[287,288,289,290],[186],[126,116],[126,36,116],[36],[292,293,294,295],[276,91],[336,91],[226,91],[91],[297,298,299,300],[352],[353],[],[],[302,303,304,305],[136],[136,226],[296],[],[307,308,309,310],[341,246,76,131],[341,246,131,76],[341,76],[131],[312,313,314,315],[191],[26],[],[],[317,318,319,320],[51,161,111,101],[51,161,111,101],[51,161,111,101],[51,161,111,101],[322,323,324,325],[141],[141],[],[],[327,328,329,330],[],[],[],[],[332,333,334,335],[354],[355],[356],[357],[337,338,339,340],[106,156,291,81],[106,156,291,81],[81],[106,156,81],[342,343,344,345],[331,161,306],[331,51,161,306],[331,51,161,306],[331],[],[],[],[],[],[],[],[],[],[],[],[]],\"text\":[\"GridWorldState(4, 1, false)\\nN:   1667\\n\",\"GridWorldState(5, 2, false)\\nN:    552\\n\",\"                       up\\nQ:  -1.28\\nN:     28\\n\",\"                     down\\nQ:   0.91\\nN:     82\\n\",\"                     left\\nQ:  -1.38\\nN:     30\\n\",\"                    right\\nQ:   2.42\\nN:    412\\n\",\"                       up\\nQ:  -0.05\\nN:    106\\n\",\"                     down\\nQ:   0.00\\nN:    120\\n\",\"                     left\\nQ:  -0.24\\nN:     98\\n\",\"                    right\\nQ:   1.98\\nN:   1343\\n\",\"GridWorldState(5, 1, false)\\nN:   1306\\n\",\"                       up\\nQ:   1.25\\nN:    119\\n\",\"                     down\\nQ:   0.00\\nN:     66\\n\",\"                     left\\nQ:   0.01\\nN:     54\\n\",\"                    right\\nQ:   2.90\\nN:   1067\\n\",\"GridWorldState(9, 8, false)\\nN:      0\\n\",\"                       up\\nQ:   0.00\\nN:      0\\n\",\"                     down\\nQ:   0.00\\nN:      0\\n\",\"                     left\\nQ:   0.00\\nN:      0\\n\",\"                    right\\nQ:   0.00\\nN:      0\\n\",\"GridWorldState(7, 9, false)\\nN:      0\\n\",\"                       up\\nQ:   0.00\\nN:      0\\n\",\"                     down\\nQ:   0.00\\nN:      0\\n\",\"                     left\\nQ:   0.00\\nN:      0\\n\",\"                    right\\nQ:   0.00\\nN:      0\\n\",\"GridWorldState(1, 5, false)\\nN:     22\\n\",\"                       up\\nQ:   0.00\\nN:      6\\n\",\"                     down\\nQ:   0.00\\nN:      2\\n\",\"                     left\\nQ:   0.00\\nN:     11\\n\",\"                    right\\nQ:   0.00\\nN:      3\\n\",\"GridWorldState(6, 2, false)\\nN:    663\\n\",\"                       up\\nQ:   3.12\\nN:    182\\n\",\"                     down\\nQ:   2.35\\nN:    100\\n\",\"                     left\\nQ:   0.32\\nN:     34\\n\",\"                    right\\nQ:   3.59\\nN:    347\\n\",\"GridWorldState(9, 6, false)\\nN:      9\\n\",\"                       up\\nQ:   1.59\\nN:      3\\n\",\"                     down\\nQ:   0.00\\nN:      2\\n\",\"                     left\\nQ:   0.00\\nN:      2\\n\",\"                    right\\nQ:   0.00\\nN:      2\\n\",\"GridWorldState(2, 5, false)\\nN:     11\\n\",\"                       up\\nQ:   0.00\\nN:      5\\n\",\"                     down\\nQ:   0.00\\nN:      2\\n\",\"                     left\\nQ:   0.00\\nN:      2\\n\",\"                    right\\nQ:   0.00\\nN:      2\\n\",\"GridWorldState(1, 4, false)\\nN:     39\\n\",\"                       up\\nQ:  -0.59\\nN:      9\\n\",\"                     down\\nQ:   0.00\\nN:      8\\n\",\"                     left\\nQ:   0.00\\nN:     14\\n\",\"                    right\\nQ:   0.00\\nN:      8\\n\",\"GridWorldState(3, 2, false)\\nN:    232\\n\",\"                       up\\nQ:  -1.11\\nN:     40\\n\",\"                     down\\nQ:   0.00\\nN:     77\\n\",\"                     left\\nQ:  -0.13\\nN:     69\\n\",\"                    right\\nQ:  -0.81\\nN:     46\\n\",\"GridWorldState(9, 1, false)\\nN:    413\\n\",\"                       up\\nQ:   5.81\\nN:    343\\n\",\"                     down\\nQ:   0.00\\nN:     19\\n\",\"                     left\\nQ:   1.09\\nN:     17\\n\",\"                    right\\nQ:   2.63\\nN:     34\\n\",\"GridWorldState(6, 1, false)\\nN:   1041\\n\",\"                       up\\nQ:   2.45\\nN:    141\\n\",\"                     down\\nQ:   0.00\\nN:     34\\n\",\"                     left\\nQ:  -0.61\\nN:     30\\n\",\"                    right\\nQ:   3.67\\nN:    836\\n\",\"GridWorldState(3, 1, false)\\nN:    506\\n\",\"                       up\\nQ:  -0.25\\nN:     76\\n\",\"                     down\\nQ:   0.00\\nN:     97\\n\",\"                     left\\nQ:  -0.55\\nN:     63\\n\",\"                    right\\nQ:   1.20\\nN:    270\\n\",\"GridWorldState(1, 1, false)\\nN:    154\\n\",\"                       up\\nQ:   0.00\\nN:     37\\n\",\"                     down\\nQ:   0.00\\nN:     39\\n\",\"                     left\\nQ:   0.00\\nN:     39\\n\",\"                    right\\nQ:   0.00\\nN:     39\\n\",\"GridWorldState(2, 4, false)\\nN:     33\\n\",\"                       up\\nQ:   0.00\\nN:     11\\n\",\"                     down\\nQ:   0.00\\nN:     10\\n\",\"                     left\\nQ:   0.00\\nN:      8\\n\",\"                    right\\nQ:  -3.30\\nN:      4\\n\",\"GridWorldState(5, 4, false)\\nN:     41\\n\",\"                       up\\nQ:  -0.24\\nN:     10\\n\",\"                     down\\nQ:  -0.42\\nN:     10\\n\",\"                     left\\nQ:  -0.30\\nN:     12\\n\",\"                    right\\nQ:  -0.59\\nN:      9\\n\",\"GridWorldState(8, 1, false)\\nN:    573\\n\",\"                       up\\nQ:   5.31\\nN:    403\\n\",\"                     down\\nQ:   0.00\\nN:     16\\n\",\"                     left\\nQ:  -0.14\\nN:     15\\n\",\"                    right\\nQ:   4.52\\nN:    139\\n\",\"GridWorldState(7, 5, false)\\nN:     40\\n\",\"                       up\\nQ:  -0.25\\nN:      7\\n\",\"                     down\\nQ:   0.95\\nN:      9\\n\",\"                     left\\nQ:   0.00\\nN:      8\\n\",\"                    right\\nQ:   0.85\\nN:     16\\n\",\"GridWorldState(7, 2, false)\\nN:    673\\n\",\"                       up\\nQ:   4.60\\nN:    256\\n\",\"                     down\\nQ:   1.58\\nN:     33\\n\",\"                     left\\nQ:   0.66\\nN:     23\\n\",\"                    right\\nQ:   4.73\\nN:    361\\n\",\"GridWorldState(1, 2, false)\\nN:    127\\n\",\"                       up\\nQ:   0.06\\nN:     33\\n\",\"                     down\\nQ:  -0.15\\nN:     29\\n\",\"                     left\\nQ:   0.00\\nN:     32\\n\",\"                    right\\nQ:   0.08\\nN:     33\\n\",\"GridWorldState(7, 4, false)\\nN:    109\\n\",\"                       up\\nQ:   1.38\\nN:     19\\n\",\"                     down\\nQ:   3.52\\nN:     46\\n\",\"                     left\\nQ:   0.52\\nN:     14\\n\",\"                    right\\nQ:   2.62\\nN:     30\\n\",\"GridWorldState(2, 1, false)\\nN:    221\\n\",\"                       up\\nQ:  -0.39\\nN:     40\\n\",\"                     down\\nQ:   0.00\\nN:     64\\n\",\"                     left\\nQ:   0.00\\nN:     56\\n\",\"                    right\\nQ:   0.32\\nN:     61\\n\",\"GridWorldState(7, 6, false)\\nN:      8\\n\",\"                       up\\nQ:   0.00\\nN:      2\\n\",\"                     down\\nQ:   0.00\\nN:      2\\n\",\"                     left\\nQ:   0.00\\nN:      2\\n\",\"                    right\\nQ:   0.00\\nN:      2\\n\",\"GridWorldState(8, 3, false)\\nN:    567\\n\",\"                       up\\nQ:   0.00\\nN:     10\\n\",\"                     down\\nQ:   1.60\\nN:     13\\n\",\"                     left\\nQ:  -0.26\\nN:     11\\n\",\"                    right\\nQ:   7.63\\nN:    533\\n\",\"GridWorldState(8, 5, false)\\nN:     41\\n\",\"                       up\\nQ:   0.00\\nN:      8\\n\",\"                     down\\nQ:   0.71\\nN:     11\\n\",\"                     left\\nQ:   0.00\\nN:     10\\n\",\"                    right\\nQ:   0.81\\nN:     12\\n\",\"GridWorldState(4, 4, false)\\nN:     18\\n\",\"                       up\\nQ:  -4.35\\nN:      3\\n\",\"                     down\\nQ:  -9.00\\nN:      2\\n\",\"                     left\\nQ:  -0.80\\nN:      6\\n\",\"                    right\\nQ:   0.00\\nN:      7\\n\",\"GridWorldState(5, 7, false)\\nN:      1\\n\",\"                       up\\nQ:   0.00\\nN:      1\\n\",\"                     down\\nQ:   0.00\\nN:      0\\n\",\"                     left\\nQ:   0.00\\nN:      0\\n\",\"                    right\\nQ:   0.00\\nN:      0\\n\",\"GridWorldState(7, 8, false)\\nN:      1\\n\",\"                       up\\nQ:   0.00\\nN:      1\\n\",\"                     down\\nQ:   0.00\\nN:      0\\n\",\"                     left\\nQ:   0.00\\nN:      0\\n\",\"                    right\\nQ:   0.00\\nN:      0\\n\",\"GridWorldState(10, 5, false)\\nN:      8\\n\",\"                       up\\nQ:   0.00\\nN:      1\\n\",\"                     down\\nQ:   2.43\\nN:      3\\n\",\"                     left\\nQ:   0.00\\nN:      2\\n\",\"                    right\\nQ:   0.00\\nN:      2\\n\",\"GridWorldState(10, 3, false)\\nN:    100\\n\",\"                       up\\nQ:   3.79\\nN:     13\\n\",\"                     down\\nQ:   4.95\\nN:     16\\n\",\"                     left\\nQ:   7.98\\nN:     63\\n\",\"                    right\\nQ:   0.00\\nN:      8\\n\",\"GridWorldState(6, 3, false)\\nN:    265\\n\",\"                       up\\nQ:  -1.13\\nN:     13\\n\",\"                     down\\nQ:   1.37\\nN:     38\\n\",\"                     left\\nQ:   0.08\\nN:     21\\n\",\"                    right\\nQ:   3.78\\nN:    193\\n\",\"GridWorldState(2, 3, false)\\nN:     74\\n\",\"                       up\\nQ:  -0.52\\nN:     17\\n\",\"                     down\\nQ:  -0.74\\nN:     16\\n\",\"                     left\\nQ:   0.00\\nN:     22\\n\",\"                    right\\nQ:  -0.35\\nN:     19\\n\",\"GridWorldState(8, 4, false)\\nN:    104\\n\",\"                       up\\nQ:   1.51\\nN:     17\\n\",\"                     down\\nQ:   3.71\\nN:     29\\n\",\"                     left\\nQ:   2.23\\nN:     22\\n\",\"                    right\\nQ:   3.51\\nN:     36\\n\",\"GridWorldState(10, 6, false)\\nN:      2\\n\",\"                       up\\nQ:   0.00\\nN:      1\\n\",\"                     down\\nQ:   0.00\\nN:      1\\n\",\"                     left\\nQ:   0.00\\nN:      0\\n\",\"                    right\\nQ:   0.00\\nN:      0\\n\",\"GridWorldState(9, 2, false)\\nN:   1765\\n\",\"                       up\\nQ:   7.96\\nN:   1668\\n\",\"                     down\\nQ:   3.23\\nN:     26\\n\",\"                     left\\nQ:   4.23\\nN:     40\\n\",\"                    right\\nQ:   3.03\\nN:     31\\n\",\"GridWorldState(3, 6, false)\\nN:      0\\n\",\"                       up\\nQ:   0.00\\nN:      0\\n\",\"                     down\\nQ:   0.00\\nN:      0\\n\",\"                     left\\nQ:   0.00\\nN:      0\\n\",\"                    right\\nQ:   0.00\\nN:      0\\n\",\"GridWorldState(8, 7, false)\\nN:      2\\n\",\"                       up\\nQ:   0.00\\nN:      1\\n\",\"                     down\\nQ:   0.00\\nN:      1\\n\",\"                     left\\nQ:   0.00\\nN:      0\\n\",\"                    right\\nQ:   0.00\\nN:      0\\n\",\"GridWorldState(1, 7, false)\\nN:      0\\n\",\"                       up\\nQ:   0.00\\nN:      0\\n\",\"                     down\\nQ:   0.00\\nN:      0\\n\",\"                     left\\nQ:   0.00\\nN:      0\\n\",\"                    right\\nQ:   0.00\\nN:      0\\n\",\"GridWorldState(7, 3, false)\\nN:    492\\n\",\"                       up\\nQ:   2.54\\nN:     41\\n\",\"                     down\\nQ:   1.19\\nN:     22\\n\",\"                     left\\nQ:  -1.39\\nN:     18\\n\",\"                    right\\nQ:   5.53\\nN:    411\\n\",\"GridWorldState(10, 2, false)\\nN:    316\\n\",\"                       up\\nQ:   5.77\\nN:    101\\n\",\"                     down\\nQ:   2.80\\nN:     23\\n\",\"                     left\\nQ:   6.41\\nN:    175\\n\",\"                    right\\nQ:   0.00\\nN:     17\\n\",\"GridWorldState(1, 3, false)\\nN:     87\\n\",\"                       up\\nQ:   0.00\\nN:     21\\n\",\"                     down\\nQ:   0.00\\nN:     23\\n\",\"                     left\\nQ:   0.00\\nN:     29\\n\",\"                    right\\nQ:  -1.19\\nN:     14\\n\",\"GridWorldState(9, 7, false)\\nN:      1\\n\",\"                       up\\nQ:   0.00\\nN:      1\\n\",\"                     down\\nQ:   0.00\\nN:      0\\n\",\"                     left\\nQ:   0.00\\nN:      0\\n\",\"                    right\\nQ:   0.00\\nN:      0\\n\",\"GridWorldState(9, 5, false)\\nN:     22\\n\",\"                       up\\nQ:   0.00\\nN:      4\\n\",\"                     down\\nQ:   2.00\\nN:     10\\n\",\"                     left\\nQ:   0.00\\nN:      4\\n\",\"                    right\\nQ:   0.00\\nN:      4\\n\",\"GridWorldState(8, 2, false)\\nN:    845\\n\",\"                       up\\nQ:   5.87\\nN:    281\\n\",\"                     down\\nQ:   2.36\\nN:     36\\n\",\"                     left\\nQ:   1.47\\nN:     21\\n\",\"                    right\\nQ:   6.28\\nN:    507\\n\",\"GridWorldState(5, 5, false)\\nN:     17\\n\",\"                       up\\nQ:   0.00\\nN:      9\\n\",\"                     down\\nQ:   0.00\\nN:      3\\n\",\"                     left\\nQ:  -4.98\\nN:      2\\n\",\"                    right\\nQ:  -0.89\\nN:      3\\n\",\"GridWorldState(10, 1, false)\\nN:    160\\n\",\"                       up\\nQ:   4.53\\nN:    100\\n\",\"                     down\\nQ:   0.00\\nN:     18\\n\",\"                     left\\nQ:   2.78\\nN:     24\\n\",\"                    right\\nQ:   0.00\\nN:     18\\n\",\"GridWorldState(4, 5, false)\\nN:      6\\n\",\"                       up\\nQ:  -4.50\\nN:      1\\n\",\"                     down\\nQ:   0.00\\nN:      3\\n\",\"                     left\\nQ:  -6.56\\nN:      1\\n\",\"                    right\\nQ:  -4.50\\nN:      1\\n\",\"GridWorldState(4, 2, false)\\nN:    416\\n\",\"                       up\\nQ:  -8.48\\nN:      6\\n\",\"                     down\\nQ:  -0.37\\nN:    121\\n\",\"                     left\\nQ:  -0.92\\nN:     80\\n\",\"                    right\\nQ:   0.30\\nN:    209\\n\",\"GridWorldState(3, 5, false)\\nN:      4\\n\",\"                       up\\nQ:  -5.90\\nN:      1\\n\",\"                     down\\nQ:   0.00\\nN:      2\\n\",\"                     left\\nQ:  -7.29\\nN:      1\\n\",\"                    right\\nQ:   0.00\\nN:      0\\n\",\"GridWorldState(9, 3, false)\\nN:   1625\\n\",\"                       up\\nQ:  10.00\\nN:    407\\n\",\"                     down\\nQ:  10.00\\nN:    406\\n\",\"                     left\\nQ:  10.00\\nN:    406\\n\",\"                    right\\nQ:  10.00\\nN:    406\\n\",\"GridWorldState(5, 3, false)\\nN:    131\\n\",\"                       up\\nQ:  -1.51\\nN:     28\\n\",\"                     down\\nQ:   0.58\\nN:     83\\n\",\"                     left\\nQ:  -4.62\\nN:      6\\n\",\"                    right\\nQ:  -3.15\\nN:     14\\n\",\"GridWorldState(7, 1, false)\\nN:    797\\n\",\"                       up\\nQ:   3.96\\nN:    208\\n\",\"                     down\\nQ:   0.00\\nN:     32\\n\",\"                     left\\nQ:  -1.34\\nN:     15\\n\",\"                    right\\nQ:   4.50\\nN:    542\\n\",\"GridWorldState(10, 4, false)\\nN:     36\\n\",\"                       up\\nQ:   0.00\\nN:      6\\n\",\"                     down\\nQ:   4.05\\nN:      8\\n\",\"                     left\\nQ:   2.86\\nN:      7\\n\",\"                    right\\nQ:   0.00\\nN:     15\\n\",\"GridWorldState(9, 4, false)\\nN:     38\\n\",\"                       up\\nQ:   2.30\\nN:      6\\n\",\"                     down\\nQ:   6.50\\nN:     18\\n\",\"                     left\\nQ:   4.08\\nN:      8\\n\",\"                    right\\nQ:   2.20\\nN:      6\\n\",\"GridWorldState(6, 6, false)\\nN:      4\\n\",\"                       up\\nQ:   0.00\\nN:      2\\n\",\"                     down\\nQ:   0.00\\nN:      1\\n\",\"                     left\\nQ:  -4.05\\nN:      1\\n\",\"                    right\\nQ:   0.00\\nN:      0\\n\",\"GridWorldState(2, 6, false)\\nN:      4\\n\",\"                       up\\nQ:   0.00\\nN:      1\\n\",\"                     down\\nQ:   0.00\\nN:      1\\n\",\"                     left\\nQ:   0.00\\nN:      2\\n\",\"                    right\\nQ:   0.00\\nN:      0\\n\",\"GridWorldState(8, 6, false)\\nN:     15\\n\",\"                       up\\nQ:   0.61\\nN:      4\\n\",\"                     down\\nQ:   0.00\\nN:      3\\n\",\"                     left\\nQ:   0.00\\nN:      4\\n\",\"                    right\\nQ:   1.08\\nN:      4\\n\",\"GridWorldState(6, 5, false)\\nN:     16\\n\",\"                       up\\nQ:  -0.91\\nN:      4\\n\",\"                     down\\nQ:   0.00\\nN:      4\\n\",\"                     left\\nQ:  -0.91\\nN:      4\\n\",\"                    right\\nQ:   0.00\\nN:      4\\n\",\"GridWorldState(4, 6, false)\\nN:      2\\n\",\"                       up\\nQ:  -5.00\\nN:      1\\n\",\"                     down\\nQ:  -5.00\\nN:      1\\n\",\"                     left\\nQ:   0.00\\nN:      0\\n\",\"                    right\\nQ:   0.00\\nN:      0\\n\",\"GridWorldState(5, 6, false)\\nN:      7\\n\",\"                       up\\nQ:  -2.95\\nN:      1\\n\",\"                     down\\nQ:   0.00\\nN:      5\\n\",\"                     left\\nQ:  -4.50\\nN:      1\\n\",\"                    right\\nQ:   0.00\\nN:      0\\n\",\"GridWorldState(3, 4, false)\\nN:     23\\n\",\"                       up\\nQ:  -3.67\\nN:      4\\n\",\"                     down\\nQ:  -1.37\\nN:      7\\n\",\"                     left\\nQ:  -1.16\\nN:      7\\n\",\"                    right\\nQ:  -3.24\\nN:      5\\n\",\"GridWorldState(1, 6, false)\\nN:      4\\n\",\"                       up\\nQ:   0.00\\nN:      1\\n\",\"                     down\\nQ:   0.00\\nN:      3\\n\",\"                     left\\nQ:   0.00\\nN:      0\\n\",\"                    right\\nQ:   0.00\\nN:      0\\n\",\"GridWorldState(2, 2, false)\\nN:    155\\n\",\"                       up\\nQ:  -0.40\\nN:     34\\n\",\"                     down\\nQ:  -0.05\\nN:     40\\n\",\"                     left\\nQ:   0.00\\nN:     40\\n\",\"                    right\\nQ:  -0.08\\nN:     41\\n\",\"GridWorldState(7, 7, false)\\nN:      2\\n\",\"                       up\\nQ:   0.00\\nN:      1\\n\",\"                     down\\nQ:   0.00\\nN:      1\\n\",\"                     left\\nQ:   0.00\\nN:      0\\n\",\"                    right\\nQ:   0.00\\nN:      0\\n\",\"GridWorldState(6, 7, false)\\nN:      0\\n\",\"                       up\\nQ:   0.00\\nN:      0\\n\",\"                     down\\nQ:   0.00\\nN:      0\\n\",\"                     left\\nQ:   0.00\\nN:      0\\n\",\"                    right\\nQ:   0.00\\nN:      0\\n\",\"GridWorldState(4, 3, false)\\nN:     79\\n\",\"                       up\\nQ: -10.00\\nN:     20\\n\",\"                     down\\nQ: -10.00\\nN:     20\\n\",\"                     left\\nQ: -10.00\\nN:     20\\n\",\"                    right\\nQ: -10.00\\nN:     19\\n\",\"GridWorldState(6, 4, false)\\nN:     55\\n\",\"                       up\\nQ:  -0.60\\nN:     11\\n\",\"                     down\\nQ:   0.57\\nN:     17\\n\",\"                     left\\nQ:  -0.56\\nN:     13\\n\",\"                    right\\nQ:  -0.05\\nN:     14\\n\",\"GridWorldState(3, 3, false)\\nN:     69\\n\",\"                       up\\nQ:  -4.56\\nN:      8\\n\",\"                     down\\nQ:  -1.34\\nN:     23\\n\",\"                     left\\nQ:  -0.39\\nN:     35\\n\",\"                    right\\nQ:  -9.00\\nN:      3\\n\",\"GridWorldState(10, 7, false)\\nN:      0\\n\",\"GridWorldState(9, 3, true)\\nN:      0\\n\",\"GridWorldState(9, 3, true)\\nN:      0\\n\",\"GridWorldState(9, 3, true)\\nN:      0\\n\",\"GridWorldState(9, 3, true)\\nN:      0\\n\",\"GridWorldState(2, 7, false)\\nN:      0\\n\",\"GridWorldState(4, 6, true)\\nN:      0\\n\",\"GridWorldState(4, 6, true)\\nN:      0\\n\",\"GridWorldState(4, 3, true)\\nN:      0\\n\",\"GridWorldState(4, 3, true)\\nN:      0\\n\",\"GridWorldState(4, 3, true)\\nN:      0\\n\",\"GridWorldState(4, 3, true)\\nN:      0\\n\"],\"tooltip\":[\"GridWorldState(4, 1, false)\\nN: 1667\\n\",\"GridWorldState(5, 2, false)\\nN: 552\\n\",\"up\\nQ: -1.2793890728250004\\nN: 28\\n\",\"down\\nQ: 0.9116536495414875\\nN: 82\\n\",\"left\\nQ: -1.38364521234687\\nN: 30\\n\",\"right\\nQ: 2.4218433334847265\\nN: 412\\n\",\"up\\nQ: -0.05072264623580763\\nN: 106\\n\",\"down\\nQ: 0.0\\nN: 120\\n\",\"left\\nQ: -0.23568040999731668\\nN: 98\\n\",\"right\\nQ: 1.9802148781745337\\nN: 1343\\n\",\"GridWorldState(5, 1, false)\\nN: 1306\\n\",\"up\\nQ: 1.2464311870170734\\nN: 119\\n\",\"down\\nQ: 0.0\\nN: 66\\n\",\"left\\nQ: 0.012753381015000034\\nN: 54\\n\",\"right\\nQ: 2.8954715423208697\\nN: 1067\\n\",\"GridWorldState(9, 8, false)\\nN: 0\\n\",\"up\\nQ: 0.0\\nN: 0\\n\",\"down\\nQ: 0.0\\nN: 0\\n\",\"left\\nQ: 0.0\\nN: 0\\n\",\"right\\nQ: 0.0\\nN: 0\\n\",\"GridWorldState(7, 9, false)\\nN: 0\\n\",\"up\\nQ: 0.0\\nN: 0\\n\",\"down\\nQ: 0.0\\nN: 0\\n\",\"left\\nQ: 0.0\\nN: 0\\n\",\"right\\nQ: 0.0\\nN: 0\\n\",\"GridWorldState(1, 5, false)\\nN: 22\\n\",\"up\\nQ: 0.0\\nN: 6\\n\",\"down\\nQ: 0.0\\nN: 2\\n\",\"left\\nQ: 0.0\\nN: 11\\n\",\"right\\nQ: 0.0\\nN: 3\\n\",\"GridWorldState(6, 2, false)\\nN: 663\\n\",\"up\\nQ: 3.1188464372164986\\nN: 182\\n\",\"down\\nQ: 2.350706165890119\\nN: 100\\n\",\"left\\nQ: 0.3172942803756176\\nN: 34\\n\",\"right\\nQ: 3.5886699641682025\\nN: 347\\n\",\"GridWorldState(9, 6, false)\\nN: 9\\n\",\"up\\nQ: 1.5943230000000002\\nN: 3\\n\",\"down\\nQ: 0.0\\nN: 2\\n\",\"left\\nQ: 0.0\\nN: 2\\n\",\"right\\nQ: 0.0\\nN: 2\\n\",\"GridWorldState(2, 5, false)\\nN: 11\\n\",\"up\\nQ: 0.0\\nN: 5\\n\",\"down\\nQ: 0.0\\nN: 2\\n\",\"left\\nQ: 0.0\\nN: 2\\n\",\"right\\nQ: 0.0\\nN: 2\\n\",\"GridWorldState(1, 4, false)\\nN: 39\\n\",\"up\\nQ: -0.5904900000000001\\nN: 9\\n\",\"down\\nQ: 0.0\\nN: 8\\n\",\"left\\nQ: 0.0\\nN: 14\\n\",\"right\\nQ: 0.0\\nN: 8\\n\",\"GridWorldState(3, 2, false)\\nN: 232\\n\",\"up\\nQ: -1.1078886235274998\\nN: 40\\n\",\"down\\nQ: 0.0028547325792269086\\nN: 77\\n\",\"left\\nQ: -0.13408470608321754\\nN: 69\\n\",\"right\\nQ: -0.8103232876145441\\nN: 46\\n\",\"GridWorldState(9, 1, false)\\nN: 413\\n\",\"up\\nQ: 5.809357999837962\\nN: 343\\n\",\"down\\nQ: 0.0\\nN: 19\\n\",\"left\\nQ: 1.087358578137\\nN: 17\\n\",\"right\\nQ: 2.632466751906176\\nN: 34\\n\",\"GridWorldState(6, 1, false)\\nN: 1041\\n\",\"up\\nQ: 2.4532668428985596\\nN: 141\\n\",\"down\\nQ: 0.0\\nN: 34\\n\",\"left\\nQ: -0.614547\\nN: 30\\n\",\"right\\nQ: 3.6725902896425184\\nN: 836\\n\",\"GridWorldState(3, 1, false)\\nN: 506\\n\",\"up\\nQ: -0.2470485566309864\\nN: 76\\n\",\"down\\nQ: 0.0\\nN: 97\\n\",\"left\\nQ: -0.5461617490506\\nN: 63\\n\",\"right\\nQ: 1.203877613068354\\nN: 270\\n\",\"GridWorldState(1, 1, false)\\nN: 154\\n\",\"up\\nQ: 0.0\\nN: 37\\n\",\"down\\nQ: 0.0\\nN: 39\\n\",\"left\\nQ: 0.0\\nN: 39\\n\",\"right\\nQ: 0.0\\nN: 39\\n\",\"GridWorldState(2, 4, false)\\nN: 33\\n\",\"up\\nQ: 0.0\\nN: 11\\n\",\"down\\nQ: 0.0\\nN: 10\\n\",\"left\\nQ: 0.0\\nN: 8\\n\",\"right\\nQ: -3.298725\\nN: 4\\n\",\"GridWorldState(5, 4, false)\\nN: 41\\n\",\"up\\nQ: -0.23914845000000007\\nN: 10\\n\",\"down\\nQ: -0.42257951099999996\\nN: 10\\n\",\"left\\nQ: -0.3037500000000001\\nN: 12\\n\",\"right\\nQ: -0.5904900000000001\\nN: 9\\n\",\"GridWorldState(8, 1, false)\\nN: 573\\n\",\"up\\nQ: 5.308893323929134\\nN: 403\\n\",\"down\\nQ: 0.0\\nN: 16\\n\",\"left\\nQ: -0.1407656474513999\\nN: 15\\n\",\"right\\nQ: 4.518211381432847\\nN: 139\\n\",\"GridWorldState(7, 5, false)\\nN: 40\\n\",\"up\\nQ: -0.24905602864285725\\nN: 7\\n\",\"down\\nQ: 0.9477000000000001\\nN: 9\\n\",\"left\\nQ: 0.0\\nN: 8\\n\",\"right\\nQ: 0.845910550125\\nN: 16\\n\",\"GridWorldState(7, 2, false)\\nN: 673\\n\",\"up\\nQ: 4.59948183598112\\nN: 256\\n\",\"down\\nQ: 1.5789193192273911\\nN: 33\\n\",\"left\\nQ: 0.6594794886913043\\nN: 23\\n\",\"right\\nQ: 4.729419438233372\\nN: 361\\n\",\"GridWorldState(1, 2, false)\\nN: 127\\n\",\"up\\nQ: 0.06239125214989365\\nN: 33\\n\",\"down\\nQ: -0.1484369689655172\\nN: 29\\n\",\"left\\nQ: 0.0\\nN: 32\\n\",\"right\\nQ: 0.07702623722209091\\nN: 33\\n\",\"GridWorldState(7, 4, false)\\nN: 109\\n\",\"up\\nQ: 1.3792551062921055\\nN: 19\\n\",\"down\\nQ: 3.5151409653456525\\nN: 46\\n\",\"left\\nQ: 0.5207142857142857\\nN: 14\\n\",\"right\\nQ: 2.61697615803\\nN: 30\\n\",\"GridWorldState(2, 1, false)\\nN: 221\\n\",\"up\\nQ: -0.3903964395764235\\nN: 40\\n\",\"down\\nQ: 0.0\\nN: 64\\n\",\"left\\nQ: 0.0\\nN: 56\\n\",\"right\\nQ: 0.3190322116000024\\nN: 61\\n\",\"GridWorldState(7, 6, false)\\nN: 8\\n\",\"up\\nQ: 0.0\\nN: 2\\n\",\"down\\nQ: 0.0\\nN: 2\\n\",\"left\\nQ: 0.0\\nN: 2\\n\",\"right\\nQ: 0.0\\nN: 2\\n\",\"GridWorldState(8, 3, false)\\nN: 567\\n\",\"up\\nQ: 0.0\\nN: 10\\n\",\"down\\nQ: 1.600753846153846\\nN: 13\\n\",\"left\\nQ: -0.2567541240736364\\nN: 11\\n\",\"right\\nQ: 7.628737872224816\\nN: 533\\n\",\"GridWorldState(8, 5, false)\\nN: 41\\n\",\"up\\nQ: 0.0\\nN: 8\\n\",\"down\\nQ: 0.7120613536363636\\nN: 11\\n\",\"left\\nQ: 0.0\\nN: 10\\n\",\"right\\nQ: 0.8149254075000001\\nN: 12\\n\",\"GridWorldState(4, 4, false)\\nN: 18\\n\",\"up\\nQ: -4.35\\nN: 3\\n\",\"down\\nQ: -9.0\\nN: 2\\n\",\"left\\nQ: -0.7971615000000002\\nN: 6\\n\",\"right\\nQ: 0.0\\nN: 7\\n\",\"GridWorldState(5, 7, false)\\nN: 1\\n\",\"up\\nQ: 0.0\\nN: 1\\n\",\"down\\nQ: 0.0\\nN: 0\\n\",\"left\\nQ: 0.0\\nN: 0\\n\",\"right\\nQ: 0.0\\nN: 0\\n\",\"GridWorldState(7, 8, false)\\nN: 1\\n\",\"up\\nQ: 0.0\\nN: 1\\n\",\"down\\nQ: 0.0\\nN: 0\\n\",\"left\\nQ: 0.0\\nN: 0\\n\",\"right\\nQ: 0.0\\nN: 0\\n\",\"GridWorldState(10, 5, false)\\nN: 8\\n\",\"up\\nQ: 0.0\\nN: 1\\n\",\"down\\nQ: 2.4299999999999997\\nN: 3\\n\",\"left\\nQ: 0.0\\nN: 2\\n\",\"right\\nQ: 0.0\\nN: 2\\n\",\"GridWorldState(10, 3, false)\\nN: 100\\n\",\"up\\nQ: 3.7863935546881917\\nN: 13\\n\",\"down\\nQ: 4.946669294895562\\nN: 16\\n\",\"left\\nQ: 7.9751201428571425\\nN: 63\\n\",\"right\\nQ: 0.0\\nN: 8\\n\",\"GridWorldState(6, 3, false)\\nN: 265\\n\",\"up\\nQ: -1.1277692307692309\\nN: 13\\n\",\"down\\nQ: 1.3700119594736844\\nN: 38\\n\",\"left\\nQ: 0.07603078806450007\\nN: 21\\n\",\"right\\nQ: 3.7772488715817136\\nN: 193\\n\",\"GridWorldState(2, 3, false)\\nN: 74\\n\",\"up\\nQ: -0.5177173177058824\\nN: 17\\n\",\"down\\nQ: -0.742213125\\nN: 16\\n\",\"left\\nQ: 0.0\\nN: 22\\n\",\"right\\nQ: -0.34531578947368424\\nN: 19\\n\",\"GridWorldState(8, 4, false)\\nN: 104\\n\",\"up\\nQ: 1.5086308605882357\\nN: 17\\n\",\"down\\nQ: 3.7074652448275858\\nN: 29\\n\",\"left\\nQ: 2.2305954499772738\\nN: 22\\n\",\"right\\nQ: 3.5070044962724998\\nN: 36\\n\",\"GridWorldState(10, 6, false)\\nN: 2\\n\",\"up\\nQ: 0.0\\nN: 1\\n\",\"down\\nQ: 0.0\\nN: 1\\n\",\"left\\nQ: 0.0\\nN: 0\\n\",\"right\\nQ: 0.0\\nN: 0\\n\",\"GridWorldState(9, 2, false)\\nN: 1765\\n\",\"up\\nQ: 7.96326659304761\\nN: 1668\\n\",\"down\\nQ: 3.232528786006826\\nN: 26\\n\",\"left\\nQ: 4.2322533091745465\\nN: 40\\n\",\"right\\nQ: 3.0318925542411117\\nN: 31\\n\",\"GridWorldState(3, 6, false)\\nN: 0\\n\",\"up\\nQ: 0.0\\nN: 0\\n\",\"down\\nQ: 0.0\\nN: 0\\n\",\"left\\nQ: 0.0\\nN: 0\\n\",\"right\\nQ: 0.0\\nN: 0\\n\",\"GridWorldState(8, 7, false)\\nN: 2\\n\",\"up\\nQ: 0.0\\nN: 1\\n\",\"down\\nQ: 0.0\\nN: 1\\n\",\"left\\nQ: 0.0\\nN: 0\\n\",\"right\\nQ: 0.0\\nN: 0\\n\",\"GridWorldState(1, 7, false)\\nN: 0\\n\",\"up\\nQ: 0.0\\nN: 0\\n\",\"down\\nQ: 0.0\\nN: 0\\n\",\"left\\nQ: 0.0\\nN: 0\\n\",\"right\\nQ: 0.0\\nN: 0\\n\",\"GridWorldState(7, 3, false)\\nN: 492\\n\",\"up\\nQ: 2.5437275464368296\\nN: 41\\n\",\"down\\nQ: 1.1948404439004547\\nN: 22\\n\",\"left\\nQ: -1.3883433250500001\\nN: 18\\n\",\"right\\nQ: 5.529589581387012\\nN: 411\\n\",\"GridWorldState(10, 2, false)\\nN: 316\\n\",\"up\\nQ: 5.765163012892654\\nN: 101\\n\",\"down\\nQ: 2.7995235609385505\\nN: 23\\n\",\"left\\nQ: 6.40604551238177\\nN: 175\\n\",\"right\\nQ: 0.0\\nN: 17\\n\",\"GridWorldState(1, 3, false)\\nN: 87\\n\",\"up\\nQ: 0.0\\nN: 21\\n\",\"down\\nQ: 0.0\\nN: 23\\n\",\"left\\nQ: 0.0\\nN: 29\\n\",\"right\\nQ: -1.1851977857142855\\nN: 14\\n\",\"GridWorldState(9, 7, false)\\nN: 1\\n\",\"up\\nQ: 0.0\\nN: 1\\n\",\"down\\nQ: 0.0\\nN: 0\\n\",\"left\\nQ: 0.0\\nN: 0\\n\",\"right\\nQ: 0.0\\nN: 0\\n\",\"GridWorldState(9, 5, false)\\nN: 22\\n\",\"up\\nQ: 0.0\\nN: 4\\n\",\"down\\nQ: 1.9975410000000005\\nN: 10\\n\",\"left\\nQ: 0.0\\nN: 4\\n\",\"right\\nQ: 0.0\\nN: 4\\n\",\"GridWorldState(8, 2, false)\\nN: 845\\n\",\"up\\nQ: 5.874873407176316\\nN: 281\\n\",\"down\\nQ: 2.3642977394614504\\nN: 36\\n\",\"left\\nQ: 1.466000514816579\\nN: 21\\n\",\"right\\nQ: 6.281690243868585\\nN: 507\\n\",\"GridWorldState(5, 5, false)\\nN: 17\\n\",\"up\\nQ: 0.0\\nN: 9\\n\",\"down\\nQ: 0.0\\nN: 3\\n\",\"left\\nQ: -4.97745\\nN: 2\\n\",\"right\\nQ: -0.8857350000000002\\nN: 3\\n\",\"GridWorldState(10, 1, false)\\nN: 160\\n\",\"up\\nQ: 4.5300113829359985\\nN: 100\\n\",\"down\\nQ: 0.0\\nN: 18\\n\",\"left\\nQ: 2.7797502037500004\\nN: 24\\n\",\"right\\nQ: 0.0\\nN: 18\\n\",\"GridWorldState(4, 5, false)\\nN: 6\\n\",\"up\\nQ: -4.5\\nN: 1\\n\",\"down\\nQ: 0.0\\nN: 3\\n\",\"left\\nQ: -6.561\\nN: 1\\n\",\"right\\nQ: -4.5\\nN: 1\\n\",\"GridWorldState(4, 2, false)\\nN: 416\\n\",\"up\\nQ: -8.48415\\nN: 6\\n\",\"down\\nQ: -0.36615333745081013\\nN: 121\\n\",\"left\\nQ: -0.9182383733045648\\nN: 80\\n\",\"right\\nQ: 0.2994774615223462\\nN: 209\\n\",\"GridWorldState(3, 5, false)\\nN: 4\\n\",\"up\\nQ: -5.904900000000001\\nN: 1\\n\",\"down\\nQ: 0.0\\nN: 2\\n\",\"left\\nQ: -7.29\\nN: 1\\n\",\"right\\nQ: 0.0\\nN: 0\\n\",\"GridWorldState(9, 3, false)\\nN: 1625\\n\",\"up\\nQ: 10.0\\nN: 407\\n\",\"down\\nQ: 10.0\\nN: 406\\n\",\"left\\nQ: 10.0\\nN: 406\\n\",\"right\\nQ: 10.0\\nN: 406\\n\",\"GridWorldState(5, 3, false)\\nN: 131\\n\",\"up\\nQ: -1.5136488678214288\\nN: 28\\n\",\"down\\nQ: 0.5815640972401808\\nN: 83\\n\",\"left\\nQ: -4.621499999999999\\nN: 6\\n\",\"right\\nQ: -3.1505738213571424\\nN: 14\\n\",\"GridWorldState(7, 1, false)\\nN: 797\\n\",\"up\\nQ: 3.9647004474626155\\nN: 208\\n\",\"down\\nQ: 0.0\\nN: 32\\n\",\"left\\nQ: -1.3386368934\\nN: 15\\n\",\"right\\nQ: 4.4951464002424135\\nN: 542\\n\",\"GridWorldState(10, 4, false)\\nN: 36\\n\",\"up\\nQ: 0.0\\nN: 6\\n\",\"down\\nQ: 4.05\\nN: 8\\n\",\"left\\nQ: 2.8581087636423\\nN: 7\\n\",\"right\\nQ: 0.0\\nN: 15\\n\",\"GridWorldState(9, 4, false)\\nN: 38\\n\",\"up\\nQ: 2.2971615000000005\\nN: 6\\n\",\"down\\nQ: 6.5\\nN: 18\\n\",\"left\\nQ: 4.082230098653625\\nN: 8\\n\",\"right\\nQ: 2.1991500000000004\\nN: 6\\n\",\"GridWorldState(6, 6, false)\\nN: 4\\n\",\"up\\nQ: 0.0\\nN: 2\\n\",\"down\\nQ: 0.0\\nN: 1\\n\",\"left\\nQ: -4.05\\nN: 1\\n\",\"right\\nQ: 0.0\\nN: 0\\n\",\"GridWorldState(2, 6, false)\\nN: 4\\n\",\"up\\nQ: 0.0\\nN: 1\\n\",\"down\\nQ: 0.0\\nN: 1\\n\",\"left\\nQ: 0.0\\nN: 2\\n\",\"right\\nQ: 0.0\\nN: 0\\n\",\"GridWorldState(8, 6, false)\\nN: 15\\n\",\"up\\nQ: 0.6075\\nN: 4\\n\",\"down\\nQ: 0.0\\nN: 3\\n\",\"left\\nQ: 0.0\\nN: 4\\n\",\"right\\nQ: 1.0761680249999999\\nN: 4\\n\",\"GridWorldState(6, 5, false)\\nN: 16\\n\",\"up\\nQ: -0.91125\\nN: 4\\n\",\"down\\nQ: 0.0\\nN: 4\\n\",\"left\\nQ: -0.9112499999999999\\nN: 4\\n\",\"right\\nQ: 0.0\\nN: 4\\n\",\"GridWorldState(4, 6, false)\\nN: 2\\n\",\"up\\nQ: -5.0\\nN: 1\\n\",\"down\\nQ: -5.0\\nN: 1\\n\",\"left\\nQ: 0.0\\nN: 0\\n\",\"right\\nQ: 0.0\\nN: 0\\n\",\"GridWorldState(5, 6, false)\\nN: 7\\n\",\"up\\nQ: -2.9524500000000007\\nN: 1\\n\",\"down\\nQ: 0.0\\nN: 5\\n\",\"left\\nQ: -4.5\\nN: 1\\n\",\"right\\nQ: 0.0\\nN: 0\\n\",\"GridWorldState(3, 4, false)\\nN: 23\\n\",\"up\\nQ: -3.66525\\nN: 4\\n\",\"down\\nQ: -1.3741545857142858\\nN: 7\\n\",\"left\\nQ: -1.157142857142857\\nN: 7\\n\",\"right\\nQ: -3.24\\nN: 5\\n\",\"GridWorldState(1, 6, false)\\nN: 4\\n\",\"up\\nQ: 0.0\\nN: 1\\n\",\"down\\nQ: 0.0\\nN: 3\\n\",\"left\\nQ: 0.0\\nN: 0\\n\",\"right\\nQ: 0.0\\nN: 0\\n\",\"GridWorldState(2, 2, false)\\nN: 155\\n\",\"up\\nQ: -0.40174984248540885\\nN: 34\\n\",\"down\\nQ: -0.04835844755282246\\nN: 40\\n\",\"left\\nQ: 0.0\\nN: 40\\n\",\"right\\nQ: -0.07681220891834292\\nN: 41\\n\",\"GridWorldState(7, 7, false)\\nN: 2\\n\",\"up\\nQ: 0.0\\nN: 1\\n\",\"down\\nQ: 0.0\\nN: 1\\n\",\"left\\nQ: 0.0\\nN: 0\\n\",\"right\\nQ: 0.0\\nN: 0\\n\",\"GridWorldState(6, 7, false)\\nN: 0\\n\",\"up\\nQ: 0.0\\nN: 0\\n\",\"down\\nQ: 0.0\\nN: 0\\n\",\"left\\nQ: 0.0\\nN: 0\\n\",\"right\\nQ: 0.0\\nN: 0\\n\",\"GridWorldState(4, 3, false)\\nN: 79\\n\",\"up\\nQ: -10.0\\nN: 20\\n\",\"down\\nQ: -10.0\\nN: 20\\n\",\"left\\nQ: -10.0\\nN: 20\\n\",\"right\\nQ: -10.0\\nN: 19\\n\",\"GridWorldState(6, 4, false)\\nN: 55\\n\",\"up\\nQ: -0.5964545454545456\\nN: 11\\n\",\"down\\nQ: 0.5719455264705883\\nN: 17\\n\",\"left\\nQ: -0.5607692307692308\\nN: 13\\n\",\"right\\nQ: -0.052071428571428435\\nN: 14\\n\",\"GridWorldState(3, 3, false)\\nN: 69\\n\",\"up\\nQ: -4.556775611250001\\nN: 8\\n\",\"down\\nQ: -1.3413945169565216\\nN: 23\\n\",\"left\\nQ: -0.3937991142857143\\nN: 35\\n\",\"right\\nQ: -9.0\\nN: 3\\n\",\"GridWorldState(10, 7, false)\\nN: 0\\n\",\"GridWorldState(9, 3, true)\\nN: 0\\n\",\"GridWorldState(9, 3, true)\\nN: 0\\n\",\"GridWorldState(9, 3, true)\\nN: 0\\n\",\"GridWorldState(9, 3, true)\\nN: 0\\n\",\"GridWorldState(2, 7, false)\\nN: 0\\n\",\"GridWorldState(4, 6, true)\\nN: 0\\n\",\"GridWorldState(4, 6, true)\\nN: 0\\n\",\"GridWorldState(4, 3, true)\\nN: 0\\n\",\"GridWorldState(4, 3, true)\\nN: 0\\n\",\"GridWorldState(4, 3, true)\\nN: 0\\n\",\"GridWorldState(4, 3, true)\\nN: 0\\n\"],\"style\":[\"\",\"\",\"stroke:#903800\",\"stroke:#744600\",\"stroke:#913700\",\"stroke:#614F00\",\"stroke:#804000\",\"stroke:#7F4000\",\"stroke:#833E00\",\"stroke:#664D00\",\"\",\"stroke:#704800\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#5B5200\",\"\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"\",\"stroke:#585400\",\"stroke:#624F00\",\"stroke:#7B4200\",\"stroke:#525700\",\"\",\"stroke:#6B4A00\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"\",\"stroke:#873C00\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"\",\"stroke:#8E3900\",\"stroke:#7F4000\",\"stroke:#813F00\",\"stroke:#8A3B00\",\"\",\"stroke:#356500\",\"stroke:#7F4000\",\"stroke:#724700\",\"stroke:#5E5100\",\"\",\"stroke:#605000\",\"stroke:#7F4000\",\"stroke:#873C00\",\"stroke:#515700\",\"\",\"stroke:#833E00\",\"stroke:#7F4000\",\"stroke:#863D00\",\"stroke:#704800\",\"\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#AA2B00\",\"\",\"stroke:#833E00\",\"stroke:#853D00\",\"stroke:#833E00\",\"stroke:#873C00\",\"\",\"stroke:#3C6200\",\"stroke:#7F4000\",\"stroke:#813F00\",\"stroke:#465D00\",\"\",\"stroke:#833E00\",\"stroke:#734600\",\"stroke:#7F4000\",\"stroke:#754500\",\"\",\"stroke:#455D00\",\"stroke:#6B4A00\",\"stroke:#774400\",\"stroke:#435E00\",\"\",\"stroke:#7F4000\",\"stroke:#813F00\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"\",\"stroke:#6E4900\",\"stroke:#535600\",\"stroke:#794300\",\"stroke:#5E5100\",\"\",\"stroke:#843E00\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7B4200\",\"\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"\",\"stroke:#7F4000\",\"stroke:#6B4A00\",\"stroke:#833E00\",\"stroke:#1E7100\",\"\",\"stroke:#7F4000\",\"stroke:#764500\",\"stroke:#7F4000\",\"stroke:#754500\",\"\",\"stroke:#B72400\",\"stroke:#F20700\",\"stroke:#8A3B00\",\"stroke:#7F4000\",\"\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"\",\"stroke:#7F4000\",\"stroke:#614F00\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"\",\"stroke:#4F5800\",\"stroke:#406000\",\"stroke:#1A7300\",\"stroke:#7F4000\",\"\",\"stroke:#8E3900\",\"stroke:#6E4900\",\"stroke:#7F4000\",\"stroke:#4F5800\",\"\",\"stroke:#863D00\",\"stroke:#893B00\",\"stroke:#7F4000\",\"stroke:#843E00\",\"\",\"stroke:#6C4A00\",\"stroke:#505800\",\"stroke:#634E00\",\"stroke:#535600\",\"\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"\",\"stroke:#1A7300\",\"stroke:#565500\",\"stroke:#4A5B00\",\"stroke:#595300\",\"\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"\",\"stroke:#5F5000\",\"stroke:#704800\",\"stroke:#913700\",\"stroke:#396300\",\"\",\"stroke:#366500\",\"stroke:#5C5200\",\"stroke:#2E6900\",\"stroke:#7F4000\",\"\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#8F3800\",\"\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"\",\"stroke:#7F4000\",\"stroke:#664D00\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"\",\"stroke:#356500\",\"stroke:#614F00\",\"stroke:#6D4900\",\"stroke:#2F6800\",\"\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#BF2000\",\"stroke:#8B3A00\",\"\",\"stroke:#465D00\",\"stroke:#7F4000\",\"stroke:#5C5200\",\"stroke:#7F4000\",\"\",\"stroke:#B92300\",\"stroke:#7F4000\",\"stroke:#D31600\",\"stroke:#B92300\",\"\",\"stroke:#EC0A00\",\"stroke:#843E00\",\"stroke:#8B3A00\",\"stroke:#7C4200\",\"\",\"stroke:#CB1A00\",\"stroke:#7F4000\",\"stroke:#DC1200\",\"stroke:#7F4000\",\"\",\"stroke:#008000\",\"stroke:#008000\",\"stroke:#008000\",\"stroke:#008000\",\"\",\"stroke:#933600\",\"stroke:#784400\",\"stroke:#BA2300\",\"stroke:#A82C00\",\"\",\"stroke:#4D5900\",\"stroke:#7F4000\",\"stroke:#913700\",\"stroke:#465D00\",\"\",\"stroke:#7F4000\",\"stroke:#4C5A00\",\"stroke:#5B5200\",\"stroke:#7F4000\",\"\",\"stroke:#624F00\",\"stroke:#2D6900\",\"stroke:#4B5A00\",\"stroke:#634E00\",\"\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#B32600\",\"stroke:#7F4000\",\"\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"\",\"stroke:#784400\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#724700\",\"\",\"stroke:#8B3A00\",\"stroke:#7F4000\",\"stroke:#8B3A00\",\"stroke:#7F4000\",\"\",\"stroke:#BF2000\",\"stroke:#BF2000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"\",\"stroke:#A52D00\",\"stroke:#7F4000\",\"stroke:#B92300\",\"stroke:#7F4000\",\"\",\"stroke:#AE2900\",\"stroke:#913700\",\"stroke:#8E3900\",\"stroke:#A92B00\",\"\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"\",\"stroke:#853D00\",\"stroke:#804000\",\"stroke:#7F4000\",\"stroke:#804000\",\"\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"\",\"stroke:#FF0000\",\"stroke:#FF0000\",\"stroke:#FF0000\",\"stroke:#FF0000\",\"\",\"stroke:#873C00\",\"stroke:#784400\",\"stroke:#873C00\",\"stroke:#804000\",\"\",\"stroke:#BA2300\",\"stroke:#913700\",\"stroke:#853D00\",\"stroke:#F20700\",\"\",\"\",\"\",\"\",\"\",\"\",\"\",\"\",\"\",\"\",\"\",\"\"],\"link_style\":[\"\",\"\",\"stroke-width:5.2138724590670105px\",\"stroke-width:8.922536101660608px\",\"stroke-width:5.396870722208659px\",\"stroke-width:20.0px\",\"stroke-width:5.6188174428825155px\",\"stroke-width:5.97836755560688px\",\"stroke-width:5.402627528753511px\",\"stroke-width:20.0px\",\"\",\"stroke-width:6.679151072108546px\",\"stroke-width:4.97416003373807px\",\"stroke-width:4.499297039752018px\",\"stroke-width:20.0px\",\"\",\"stroke-width:NaNpx\",\"stroke-width:NaNpx\",\"stroke-width:NaNpx\",\"stroke-width:NaNpx\",\"\",\"stroke-width:NaNpx\",\"stroke-width:NaNpx\",\"stroke-width:NaNpx\",\"stroke-width:NaNpx\",\"\",\"stroke-width:14.770978917519928px\",\"stroke-width:8.528028654224418px\",\"stroke-width:20.0px\",\"stroke-width:10.44465935734187px\",\"\",\"stroke-width:14.484414758400552px\",\"stroke-width:10.736562541900378px\",\"stroke-width:6.2604379697819095px\",\"stroke-width:20.0px\",\"\",\"stroke-width:20.0px\",\"stroke-width:16.32993161855452px\",\"stroke-width:16.32993161855452px\",\"stroke-width:16.32993161855452px\",\"\",\"stroke-width:20.0px\",\"stroke-width:12.649110640673518px\",\"stroke-width:12.649110640673518px\",\"stroke-width:12.649110640673518px\",\"\",\"stroke-width:16.035674514745462px\",\"stroke-width:15.118578920369089px\",\"stroke-width:20.0px\",\"stroke-width:15.118578920369089px\",\"\",\"stroke-width:14.414999403128943px\",\"stroke-width:20.0px\",\"stroke-width:18.932552877030567px\",\"stroke-width:15.458364692328841px\",\"\",\"stroke-width:20.0px\",\"stroke-width:4.70716840598808px\",\"stroke-width:4.452535556994169px\",\"stroke-width:6.296836171649597px\",\"\",\"stroke-width:8.213654170497799px\",\"stroke-width:4.033353763771789px\",\"stroke-width:3.788676152120412px\",\"stroke-width:20.0px\",\"\",\"stroke-width:10.610965676722952px\",\"stroke-width:11.987647963787714px\",\"stroke-width:9.66091783079296px\",\"stroke-width:20.0px\",\"\",\"stroke-width:19.480430680228288px\",\"stroke-width:20.0px\",\"stroke-width:20.0px\",\"stroke-width:20.0px\",\"\",\"stroke-width:20.0px\",\"stroke-width:19.06925178491185px\",\"stroke-width:17.056057308448835px\",\"stroke-width:12.060453783110546px\",\"\",\"stroke-width:18.257418583505537px\",\"stroke-width:18.257418583505537px\",\"stroke-width:20.0px\",\"stroke-width:17.32050807568877px\",\"\",\"stroke-width:20.0px\",\"stroke-width:3.985083851093743px\",\"stroke-width:3.8585408471315463px\",\"stroke-width:11.745861422045882px\",\"\",\"stroke-width:13.228756555322953px\",\"stroke-width:15.0px\",\"stroke-width:14.142135623730951px\",\"stroke-width:20.0px\",\"\",\"stroke-width:16.842105263157894px\",\"stroke-width:6.046908048987398px\",\"stroke-width:5.048243708750231px\",\"stroke-width:20.0px\",\"\",\"stroke-width:20.0px\",\"stroke-width:18.74873733122184px\",\"stroke-width:19.69463855669324px\",\"stroke-width:20.0px\",\"\",\"stroke-width:12.853691738343029px\",\"stroke-width:20.0px\",\"stroke-width:11.03354568734741px\",\"stroke-width:16.151457061744964px\",\"\",\"stroke-width:15.811388300841898px\",\"stroke-width:20.0px\",\"stroke-width:18.708286933869708px\",\"stroke-width:19.525624189766635px\",\"\",\"stroke-width:20.0px\",\"stroke-width:20.0px\",\"stroke-width:20.0px\",\"stroke-width:20.0px\",\"\",\"stroke-width:2.7394690053949686px\",\"stroke-width:3.1234752377721215px\",\"stroke-width:2.873179332146128px\",\"stroke-width:20.0px\",\"\",\"stroke-width:16.32993161855452px\",\"stroke-width:19.148542155126762px\",\"stroke-width:18.257418583505537px\",\"stroke-width:20.0px\",\"\",\"stroke-width:13.09307341415954px\",\"stroke-width:10.690449676496975px\",\"stroke-width:18.516401995451027px\",\"stroke-width:20.0px\",\"\",\"stroke-width:20.0px\",\"stroke-width:0.0px\",\"stroke-width:0.0px\",\"stroke-width:0.0px\",\"\",\"stroke-width:20.0px\",\"stroke-width:0.0px\",\"stroke-width:0.0px\",\"stroke-width:0.0px\",\"\",\"stroke-width:11.547005383792515px\",\"stroke-width:20.0px\",\"stroke-width:16.32993161855452px\",\"stroke-width:16.32993161855452px\",\"\",\"stroke-width:9.085135251589959px\",\"stroke-width:10.079052613579393px\",\"stroke-width:20.0px\",\"stroke-width:7.126966450997983px\",\"\",\"stroke-width:5.190665196422049px\",\"stroke-width:8.874484586945758px\",\"stroke-width:6.597220328454717px\",\"stroke-width:20.0px\",\"\",\"stroke-width:17.580981459830653px\",\"stroke-width:17.056057308448835px\",\"stroke-width:20.0px\",\"stroke-width:18.586407545691703px\",\"\",\"stroke-width:13.743685418725535px\",\"stroke-width:17.950549357115012px\",\"stroke-width:15.634719199411434px\",\"stroke-width:20.0px\",\"\",\"stroke-width:20.0px\",\"stroke-width:20.0px\",\"stroke-width:0.0px\",\"stroke-width:0.0px\",\"\",\"stroke-width:20.0px\",\"stroke-width:2.4970005988000232px\",\"stroke-width:3.0971480654125543px\",\"stroke-width:2.726545939058017px\",\"\",\"stroke-width:NaNpx\",\"stroke-width:NaNpx\",\"stroke-width:NaNpx\",\"stroke-width:NaNpx\",\"\",\"stroke-width:20.0px\",\"stroke-width:20.0px\",\"stroke-width:0.0px\",\"stroke-width:0.0px\",\"\",\"stroke-width:NaNpx\",\"stroke-width:NaNpx\",\"stroke-width:NaNpx\",\"stroke-width:NaNpx\",\"\",\"stroke-width:6.31685652829212px\",\"stroke-width:4.627222948390527px\",\"stroke-width:4.185480638490935px\",\"stroke-width:20.0px\",\"\",\"stroke-width:15.19398377178095px\",\"stroke-width:7.250615737399726px\",\"stroke-width:20.0px\",\"stroke-width:6.233549779791837px\",\"\",\"stroke-width:17.019258867935264px\",\"stroke-width:17.811271131234424px\",\"stroke-width:20.0px\",\"stroke-width:13.896166675593024px\",\"\",\"stroke-width:20.0px\",\"stroke-width:0.0px\",\"stroke-width:0.0px\",\"stroke-width:0.0px\",\"\",\"stroke-width:12.649110640673518px\",\"stroke-width:20.0px\",\"stroke-width:12.649110640673518px\",\"stroke-width:12.649110640673518px\",\"\",\"stroke-width:14.889467836879975px\",\"stroke-width:5.32938710021193px\",\"stroke-width:4.070386632407063px\",\"stroke-width:20.0px\",\"\",\"stroke-width:20.0px\",\"stroke-width:11.547005383792515px\",\"stroke-width:9.428090415820634px\",\"stroke-width:11.547005383792515px\",\"\",\"stroke-width:20.0px\",\"stroke-width:8.48528137423857px\",\"stroke-width:9.797958971132712px\",\"stroke-width:8.48528137423857px\",\"\",\"stroke-width:11.547005383792515px\",\"stroke-width:20.0px\",\"stroke-width:11.547005383792515px\",\"stroke-width:11.547005383792515px\",\"\",\"stroke-width:3.3886949683494305px\",\"stroke-width:15.217718205053643px\",\"stroke-width:12.37376449779492px\",\"stroke-width:20.0px\",\"\",\"stroke-width:14.142135623730951px\",\"stroke-width:20.0px\",\"stroke-width:14.142135623730951px\",\"stroke-width:0.0px\",\"\",\"stroke-width:20.0px\",\"stroke-width:19.975414864708043px\",\"stroke-width:19.975414864708043px\",\"stroke-width:19.975414864708043px\",\"\",\"stroke-width:11.616357391030313px\",\"stroke-width:20.0px\",\"stroke-width:5.377328579378649px\",\"stroke-width:8.214005083884006px\",\"\",\"stroke-width:12.389735068004908px\",\"stroke-width:4.859653913846296px\",\"stroke-width:3.327177587852354px\",\"stroke-width:20.0px\",\"\",\"stroke-width:12.649110640673518px\",\"stroke-width:14.605934866804429px\",\"stroke-width:13.662601021279464px\",\"stroke-width:20.0px\",\"\",\"stroke-width:11.547005383792515px\",\"stroke-width:20.0px\",\"stroke-width:13.333333333333332px\",\"stroke-width:11.547005383792515px\",\"\",\"stroke-width:20.0px\",\"stroke-width:14.142135623730951px\",\"stroke-width:14.142135623730951px\",\"stroke-width:0.0px\",\"\",\"stroke-width:14.142135623730951px\",\"stroke-width:14.142135623730951px\",\"stroke-width:20.0px\",\"stroke-width:0.0px\",\"\",\"stroke-width:20.0px\",\"stroke-width:17.32050807568877px\",\"stroke-width:20.0px\",\"stroke-width:20.0px\",\"\",\"stroke-width:20.0px\",\"stroke-width:20.0px\",\"stroke-width:20.0px\",\"stroke-width:20.0px\",\"\",\"stroke-width:20.0px\",\"stroke-width:20.0px\",\"stroke-width:0.0px\",\"stroke-width:0.0px\",\"\",\"stroke-width:8.94427190999916px\",\"stroke-width:20.0px\",\"stroke-width:8.94427190999916px\",\"stroke-width:0.0px\",\"\",\"stroke-width:15.118578920369089px\",\"stroke-width:20.0px\",\"stroke-width:20.0px\",\"stroke-width:16.903085094570333px\",\"\",\"stroke-width:11.547005383792515px\",\"stroke-width:20.0px\",\"stroke-width:0.0px\",\"stroke-width:0.0px\",\"\",\"stroke-width:18.212833856189725px\",\"stroke-width:19.754591932991794px\",\"stroke-width:19.754591932991794px\",\"stroke-width:20.0px\",\"\",\"stroke-width:20.0px\",\"stroke-width:20.0px\",\"stroke-width:0.0px\",\"stroke-width:0.0px\",\"\",\"stroke-width:NaNpx\",\"stroke-width:NaNpx\",\"stroke-width:NaNpx\",\"stroke-width:NaNpx\",\"\",\"stroke-width:20.0px\",\"stroke-width:20.0px\",\"stroke-width:20.0px\",\"stroke-width:19.493588689617926px\",\"\",\"stroke-width:16.087993330796873px\",\"stroke-width:20.0px\",\"stroke-width:17.48949264390412px\",\"stroke-width:18.149704259460602px\",\"\",\"stroke-width:9.561828874675149px\",\"stroke-width:16.21286966755555px\",\"stroke-width:20.0px\",\"stroke-width:5.855400437691199px\",\"\",\"\",\"\",\"\",\"\",\"\",\"\",\"\",\"\",\"\",\"\",\"\"],\"title\":\"MCTS tree\",\"options\":{\"tooltip\":[\"GridWorldState(4, 1, false)\\nN: 1667\\n\",\"GridWorldState(5, 2, false)\\nN: 552\\n\",\"up\\nQ: -1.2793890728250004\\nN: 28\\n\",\"down\\nQ: 0.9116536495414875\\nN: 82\\n\",\"left\\nQ: -1.38364521234687\\nN: 30\\n\",\"right\\nQ: 2.4218433334847265\\nN: 412\\n\",\"up\\nQ: -0.05072264623580763\\nN: 106\\n\",\"down\\nQ: 0.0\\nN: 120\\n\",\"left\\nQ: -0.23568040999731668\\nN: 98\\n\",\"right\\nQ: 1.9802148781745337\\nN: 1343\\n\",\"GridWorldState(5, 1, false)\\nN: 1306\\n\",\"up\\nQ: 1.2464311870170734\\nN: 119\\n\",\"down\\nQ: 0.0\\nN: 66\\n\",\"left\\nQ: 0.012753381015000034\\nN: 54\\n\",\"right\\nQ: 2.8954715423208697\\nN: 1067\\n\",\"GridWorldState(9, 8, false)\\nN: 0\\n\",\"up\\nQ: 0.0\\nN: 0\\n\",\"down\\nQ: 0.0\\nN: 0\\n\",\"left\\nQ: 0.0\\nN: 0\\n\",\"right\\nQ: 0.0\\nN: 0\\n\",\"GridWorldState(7, 9, false)\\nN: 0\\n\",\"up\\nQ: 0.0\\nN: 0\\n\",\"down\\nQ: 0.0\\nN: 0\\n\",\"left\\nQ: 0.0\\nN: 0\\n\",\"right\\nQ: 0.0\\nN: 0\\n\",\"GridWorldState(1, 5, false)\\nN: 22\\n\",\"up\\nQ: 0.0\\nN: 6\\n\",\"down\\nQ: 0.0\\nN: 2\\n\",\"left\\nQ: 0.0\\nN: 11\\n\",\"right\\nQ: 0.0\\nN: 3\\n\",\"GridWorldState(6, 2, false)\\nN: 663\\n\",\"up\\nQ: 3.1188464372164986\\nN: 182\\n\",\"down\\nQ: 2.350706165890119\\nN: 100\\n\",\"left\\nQ: 0.3172942803756176\\nN: 34\\n\",\"right\\nQ: 3.5886699641682025\\nN: 347\\n\",\"GridWorldState(9, 6, false)\\nN: 9\\n\",\"up\\nQ: 1.5943230000000002\\nN: 3\\n\",\"down\\nQ: 0.0\\nN: 2\\n\",\"left\\nQ: 0.0\\nN: 2\\n\",\"right\\nQ: 0.0\\nN: 2\\n\",\"GridWorldState(2, 5, false)\\nN: 11\\n\",\"up\\nQ: 0.0\\nN: 5\\n\",\"down\\nQ: 0.0\\nN: 2\\n\",\"left\\nQ: 0.0\\nN: 2\\n\",\"right\\nQ: 0.0\\nN: 2\\n\",\"GridWorldState(1, 4, false)\\nN: 39\\n\",\"up\\nQ: -0.5904900000000001\\nN: 9\\n\",\"down\\nQ: 0.0\\nN: 8\\n\",\"left\\nQ: 0.0\\nN: 14\\n\",\"right\\nQ: 0.0\\nN: 8\\n\",\"GridWorldState(3, 2, false)\\nN: 232\\n\",\"up\\nQ: -1.1078886235274998\\nN: 40\\n\",\"down\\nQ: 0.0028547325792269086\\nN: 77\\n\",\"left\\nQ: -0.13408470608321754\\nN: 69\\n\",\"right\\nQ: -0.8103232876145441\\nN: 46\\n\",\"GridWorldState(9, 1, false)\\nN: 413\\n\",\"up\\nQ: 5.809357999837962\\nN: 343\\n\",\"down\\nQ: 0.0\\nN: 19\\n\",\"left\\nQ: 1.087358578137\\nN: 17\\n\",\"right\\nQ: 2.632466751906176\\nN: 34\\n\",\"GridWorldState(6, 1, false)\\nN: 1041\\n\",\"up\\nQ: 2.4532668428985596\\nN: 141\\n\",\"down\\nQ: 0.0\\nN: 34\\n\",\"left\\nQ: -0.614547\\nN: 30\\n\",\"right\\nQ: 3.6725902896425184\\nN: 836\\n\",\"GridWorldState(3, 1, false)\\nN: 506\\n\",\"up\\nQ: -0.2470485566309864\\nN: 76\\n\",\"down\\nQ: 0.0\\nN: 97\\n\",\"left\\nQ: -0.5461617490506\\nN: 63\\n\",\"right\\nQ: 1.203877613068354\\nN: 270\\n\",\"GridWorldState(1, 1, false)\\nN: 154\\n\",\"up\\nQ: 0.0\\nN: 37\\n\",\"down\\nQ: 0.0\\nN: 39\\n\",\"left\\nQ: 0.0\\nN: 39\\n\",\"right\\nQ: 0.0\\nN: 39\\n\",\"GridWorldState(2, 4, false)\\nN: 33\\n\",\"up\\nQ: 0.0\\nN: 11\\n\",\"down\\nQ: 0.0\\nN: 10\\n\",\"left\\nQ: 0.0\\nN: 8\\n\",\"right\\nQ: -3.298725\\nN: 4\\n\",\"GridWorldState(5, 4, false)\\nN: 41\\n\",\"up\\nQ: -0.23914845000000007\\nN: 10\\n\",\"down\\nQ: -0.42257951099999996\\nN: 10\\n\",\"left\\nQ: -0.3037500000000001\\nN: 12\\n\",\"right\\nQ: -0.5904900000000001\\nN: 9\\n\",\"GridWorldState(8, 1, false)\\nN: 573\\n\",\"up\\nQ: 5.308893323929134\\nN: 403\\n\",\"down\\nQ: 0.0\\nN: 16\\n\",\"left\\nQ: -0.1407656474513999\\nN: 15\\n\",\"right\\nQ: 4.518211381432847\\nN: 139\\n\",\"GridWorldState(7, 5, false)\\nN: 40\\n\",\"up\\nQ: -0.24905602864285725\\nN: 7\\n\",\"down\\nQ: 0.9477000000000001\\nN: 9\\n\",\"left\\nQ: 0.0\\nN: 8\\n\",\"right\\nQ: 0.845910550125\\nN: 16\\n\",\"GridWorldState(7, 2, false)\\nN: 673\\n\",\"up\\nQ: 4.59948183598112\\nN: 256\\n\",\"down\\nQ: 1.5789193192273911\\nN: 33\\n\",\"left\\nQ: 0.6594794886913043\\nN: 23\\n\",\"right\\nQ: 4.729419438233372\\nN: 361\\n\",\"GridWorldState(1, 2, false)\\nN: 127\\n\",\"up\\nQ: 0.06239125214989365\\nN: 33\\n\",\"down\\nQ: -0.1484369689655172\\nN: 29\\n\",\"left\\nQ: 0.0\\nN: 32\\n\",\"right\\nQ: 0.07702623722209091\\nN: 33\\n\",\"GridWorldState(7, 4, false)\\nN: 109\\n\",\"up\\nQ: 1.3792551062921055\\nN: 19\\n\",\"down\\nQ: 3.5151409653456525\\nN: 46\\n\",\"left\\nQ: 0.5207142857142857\\nN: 14\\n\",\"right\\nQ: 2.61697615803\\nN: 30\\n\",\"GridWorldState(2, 1, false)\\nN: 221\\n\",\"up\\nQ: -0.3903964395764235\\nN: 40\\n\",\"down\\nQ: 0.0\\nN: 64\\n\",\"left\\nQ: 0.0\\nN: 56\\n\",\"right\\nQ: 0.3190322116000024\\nN: 61\\n\",\"GridWorldState(7, 6, false)\\nN: 8\\n\",\"up\\nQ: 0.0\\nN: 2\\n\",\"down\\nQ: 0.0\\nN: 2\\n\",\"left\\nQ: 0.0\\nN: 2\\n\",\"right\\nQ: 0.0\\nN: 2\\n\",\"GridWorldState(8, 3, false)\\nN: 567\\n\",\"up\\nQ: 0.0\\nN: 10\\n\",\"down\\nQ: 1.600753846153846\\nN: 13\\n\",\"left\\nQ: -0.2567541240736364\\nN: 11\\n\",\"right\\nQ: 7.628737872224816\\nN: 533\\n\",\"GridWorldState(8, 5, false)\\nN: 41\\n\",\"up\\nQ: 0.0\\nN: 8\\n\",\"down\\nQ: 0.7120613536363636\\nN: 11\\n\",\"left\\nQ: 0.0\\nN: 10\\n\",\"right\\nQ: 0.8149254075000001\\nN: 12\\n\",\"GridWorldState(4, 4, false)\\nN: 18\\n\",\"up\\nQ: -4.35\\nN: 3\\n\",\"down\\nQ: -9.0\\nN: 2\\n\",\"left\\nQ: -0.7971615000000002\\nN: 6\\n\",\"right\\nQ: 0.0\\nN: 7\\n\",\"GridWorldState(5, 7, false)\\nN: 1\\n\",\"up\\nQ: 0.0\\nN: 1\\n\",\"down\\nQ: 0.0\\nN: 0\\n\",\"left\\nQ: 0.0\\nN: 0\\n\",\"right\\nQ: 0.0\\nN: 0\\n\",\"GridWorldState(7, 8, false)\\nN: 1\\n\",\"up\\nQ: 0.0\\nN: 1\\n\",\"down\\nQ: 0.0\\nN: 0\\n\",\"left\\nQ: 0.0\\nN: 0\\n\",\"right\\nQ: 0.0\\nN: 0\\n\",\"GridWorldState(10, 5, false)\\nN: 8\\n\",\"up\\nQ: 0.0\\nN: 1\\n\",\"down\\nQ: 2.4299999999999997\\nN: 3\\n\",\"left\\nQ: 0.0\\nN: 2\\n\",\"right\\nQ: 0.0\\nN: 2\\n\",\"GridWorldState(10, 3, false)\\nN: 100\\n\",\"up\\nQ: 3.7863935546881917\\nN: 13\\n\",\"down\\nQ: 4.946669294895562\\nN: 16\\n\",\"left\\nQ: 7.9751201428571425\\nN: 63\\n\",\"right\\nQ: 0.0\\nN: 8\\n\",\"GridWorldState(6, 3, false)\\nN: 265\\n\",\"up\\nQ: -1.1277692307692309\\nN: 13\\n\",\"down\\nQ: 1.3700119594736844\\nN: 38\\n\",\"left\\nQ: 0.07603078806450007\\nN: 21\\n\",\"right\\nQ: 3.7772488715817136\\nN: 193\\n\",\"GridWorldState(2, 3, false)\\nN: 74\\n\",\"up\\nQ: -0.5177173177058824\\nN: 17\\n\",\"down\\nQ: -0.742213125\\nN: 16\\n\",\"left\\nQ: 0.0\\nN: 22\\n\",\"right\\nQ: -0.34531578947368424\\nN: 19\\n\",\"GridWorldState(8, 4, false)\\nN: 104\\n\",\"up\\nQ: 1.5086308605882357\\nN: 17\\n\",\"down\\nQ: 3.7074652448275858\\nN: 29\\n\",\"left\\nQ: 2.2305954499772738\\nN: 22\\n\",\"right\\nQ: 3.5070044962724998\\nN: 36\\n\",\"GridWorldState(10, 6, false)\\nN: 2\\n\",\"up\\nQ: 0.0\\nN: 1\\n\",\"down\\nQ: 0.0\\nN: 1\\n\",\"left\\nQ: 0.0\\nN: 0\\n\",\"right\\nQ: 0.0\\nN: 0\\n\",\"GridWorldState(9, 2, false)\\nN: 1765\\n\",\"up\\nQ: 7.96326659304761\\nN: 1668\\n\",\"down\\nQ: 3.232528786006826\\nN: 26\\n\",\"left\\nQ: 4.2322533091745465\\nN: 40\\n\",\"right\\nQ: 3.0318925542411117\\nN: 31\\n\",\"GridWorldState(3, 6, false)\\nN: 0\\n\",\"up\\nQ: 0.0\\nN: 0\\n\",\"down\\nQ: 0.0\\nN: 0\\n\",\"left\\nQ: 0.0\\nN: 0\\n\",\"right\\nQ: 0.0\\nN: 0\\n\",\"GridWorldState(8, 7, false)\\nN: 2\\n\",\"up\\nQ: 0.0\\nN: 1\\n\",\"down\\nQ: 0.0\\nN: 1\\n\",\"left\\nQ: 0.0\\nN: 0\\n\",\"right\\nQ: 0.0\\nN: 0\\n\",\"GridWorldState(1, 7, false)\\nN: 0\\n\",\"up\\nQ: 0.0\\nN: 0\\n\",\"down\\nQ: 0.0\\nN: 0\\n\",\"left\\nQ: 0.0\\nN: 0\\n\",\"right\\nQ: 0.0\\nN: 0\\n\",\"GridWorldState(7, 3, false)\\nN: 492\\n\",\"up\\nQ: 2.5437275464368296\\nN: 41\\n\",\"down\\nQ: 1.1948404439004547\\nN: 22\\n\",\"left\\nQ: -1.3883433250500001\\nN: 18\\n\",\"right\\nQ: 5.529589581387012\\nN: 411\\n\",\"GridWorldState(10, 2, false)\\nN: 316\\n\",\"up\\nQ: 5.765163012892654\\nN: 101\\n\",\"down\\nQ: 2.7995235609385505\\nN: 23\\n\",\"left\\nQ: 6.40604551238177\\nN: 175\\n\",\"right\\nQ: 0.0\\nN: 17\\n\",\"GridWorldState(1, 3, false)\\nN: 87\\n\",\"up\\nQ: 0.0\\nN: 21\\n\",\"down\\nQ: 0.0\\nN: 23\\n\",\"left\\nQ: 0.0\\nN: 29\\n\",\"right\\nQ: -1.1851977857142855\\nN: 14\\n\",\"GridWorldState(9, 7, false)\\nN: 1\\n\",\"up\\nQ: 0.0\\nN: 1\\n\",\"down\\nQ: 0.0\\nN: 0\\n\",\"left\\nQ: 0.0\\nN: 0\\n\",\"right\\nQ: 0.0\\nN: 0\\n\",\"GridWorldState(9, 5, false)\\nN: 22\\n\",\"up\\nQ: 0.0\\nN: 4\\n\",\"down\\nQ: 1.9975410000000005\\nN: 10\\n\",\"left\\nQ: 0.0\\nN: 4\\n\",\"right\\nQ: 0.0\\nN: 4\\n\",\"GridWorldState(8, 2, false)\\nN: 845\\n\",\"up\\nQ: 5.874873407176316\\nN: 281\\n\",\"down\\nQ: 2.3642977394614504\\nN: 36\\n\",\"left\\nQ: 1.466000514816579\\nN: 21\\n\",\"right\\nQ: 6.281690243868585\\nN: 507\\n\",\"GridWorldState(5, 5, false)\\nN: 17\\n\",\"up\\nQ: 0.0\\nN: 9\\n\",\"down\\nQ: 0.0\\nN: 3\\n\",\"left\\nQ: -4.97745\\nN: 2\\n\",\"right\\nQ: -0.8857350000000002\\nN: 3\\n\",\"GridWorldState(10, 1, false)\\nN: 160\\n\",\"up\\nQ: 4.5300113829359985\\nN: 100\\n\",\"down\\nQ: 0.0\\nN: 18\\n\",\"left\\nQ: 2.7797502037500004\\nN: 24\\n\",\"right\\nQ: 0.0\\nN: 18\\n\",\"GridWorldState(4, 5, false)\\nN: 6\\n\",\"up\\nQ: -4.5\\nN: 1\\n\",\"down\\nQ: 0.0\\nN: 3\\n\",\"left\\nQ: -6.561\\nN: 1\\n\",\"right\\nQ: -4.5\\nN: 1\\n\",\"GridWorldState(4, 2, false)\\nN: 416\\n\",\"up\\nQ: -8.48415\\nN: 6\\n\",\"down\\nQ: -0.36615333745081013\\nN: 121\\n\",\"left\\nQ: -0.9182383733045648\\nN: 80\\n\",\"right\\nQ: 0.2994774615223462\\nN: 209\\n\",\"GridWorldState(3, 5, false)\\nN: 4\\n\",\"up\\nQ: -5.904900000000001\\nN: 1\\n\",\"down\\nQ: 0.0\\nN: 2\\n\",\"left\\nQ: -7.29\\nN: 1\\n\",\"right\\nQ: 0.0\\nN: 0\\n\",\"GridWorldState(9, 3, false)\\nN: 1625\\n\",\"up\\nQ: 10.0\\nN: 407\\n\",\"down\\nQ: 10.0\\nN: 406\\n\",\"left\\nQ: 10.0\\nN: 406\\n\",\"right\\nQ: 10.0\\nN: 406\\n\",\"GridWorldState(5, 3, false)\\nN: 131\\n\",\"up\\nQ: -1.5136488678214288\\nN: 28\\n\",\"down\\nQ: 0.5815640972401808\\nN: 83\\n\",\"left\\nQ: -4.621499999999999\\nN: 6\\n\",\"right\\nQ: -3.1505738213571424\\nN: 14\\n\",\"GridWorldState(7, 1, false)\\nN: 797\\n\",\"up\\nQ: 3.9647004474626155\\nN: 208\\n\",\"down\\nQ: 0.0\\nN: 32\\n\",\"left\\nQ: -1.3386368934\\nN: 15\\n\",\"right\\nQ: 4.4951464002424135\\nN: 542\\n\",\"GridWorldState(10, 4, false)\\nN: 36\\n\",\"up\\nQ: 0.0\\nN: 6\\n\",\"down\\nQ: 4.05\\nN: 8\\n\",\"left\\nQ: 2.8581087636423\\nN: 7\\n\",\"right\\nQ: 0.0\\nN: 15\\n\",\"GridWorldState(9, 4, false)\\nN: 38\\n\",\"up\\nQ: 2.2971615000000005\\nN: 6\\n\",\"down\\nQ: 6.5\\nN: 18\\n\",\"left\\nQ: 4.082230098653625\\nN: 8\\n\",\"right\\nQ: 2.1991500000000004\\nN: 6\\n\",\"GridWorldState(6, 6, false)\\nN: 4\\n\",\"up\\nQ: 0.0\\nN: 2\\n\",\"down\\nQ: 0.0\\nN: 1\\n\",\"left\\nQ: -4.05\\nN: 1\\n\",\"right\\nQ: 0.0\\nN: 0\\n\",\"GridWorldState(2, 6, false)\\nN: 4\\n\",\"up\\nQ: 0.0\\nN: 1\\n\",\"down\\nQ: 0.0\\nN: 1\\n\",\"left\\nQ: 0.0\\nN: 2\\n\",\"right\\nQ: 0.0\\nN: 0\\n\",\"GridWorldState(8, 6, false)\\nN: 15\\n\",\"up\\nQ: 0.6075\\nN: 4\\n\",\"down\\nQ: 0.0\\nN: 3\\n\",\"left\\nQ: 0.0\\nN: 4\\n\",\"right\\nQ: 1.0761680249999999\\nN: 4\\n\",\"GridWorldState(6, 5, false)\\nN: 16\\n\",\"up\\nQ: -0.91125\\nN: 4\\n\",\"down\\nQ: 0.0\\nN: 4\\n\",\"left\\nQ: -0.9112499999999999\\nN: 4\\n\",\"right\\nQ: 0.0\\nN: 4\\n\",\"GridWorldState(4, 6, false)\\nN: 2\\n\",\"up\\nQ: -5.0\\nN: 1\\n\",\"down\\nQ: -5.0\\nN: 1\\n\",\"left\\nQ: 0.0\\nN: 0\\n\",\"right\\nQ: 0.0\\nN: 0\\n\",\"GridWorldState(5, 6, false)\\nN: 7\\n\",\"up\\nQ: -2.9524500000000007\\nN: 1\\n\",\"down\\nQ: 0.0\\nN: 5\\n\",\"left\\nQ: -4.5\\nN: 1\\n\",\"right\\nQ: 0.0\\nN: 0\\n\",\"GridWorldState(3, 4, false)\\nN: 23\\n\",\"up\\nQ: -3.66525\\nN: 4\\n\",\"down\\nQ: -1.3741545857142858\\nN: 7\\n\",\"left\\nQ: -1.157142857142857\\nN: 7\\n\",\"right\\nQ: -3.24\\nN: 5\\n\",\"GridWorldState(1, 6, false)\\nN: 4\\n\",\"up\\nQ: 0.0\\nN: 1\\n\",\"down\\nQ: 0.0\\nN: 3\\n\",\"left\\nQ: 0.0\\nN: 0\\n\",\"right\\nQ: 0.0\\nN: 0\\n\",\"GridWorldState(2, 2, false)\\nN: 155\\n\",\"up\\nQ: -0.40174984248540885\\nN: 34\\n\",\"down\\nQ: -0.04835844755282246\\nN: 40\\n\",\"left\\nQ: 0.0\\nN: 40\\n\",\"right\\nQ: -0.07681220891834292\\nN: 41\\n\",\"GridWorldState(7, 7, false)\\nN: 2\\n\",\"up\\nQ: 0.0\\nN: 1\\n\",\"down\\nQ: 0.0\\nN: 1\\n\",\"left\\nQ: 0.0\\nN: 0\\n\",\"right\\nQ: 0.0\\nN: 0\\n\",\"GridWorldState(6, 7, false)\\nN: 0\\n\",\"up\\nQ: 0.0\\nN: 0\\n\",\"down\\nQ: 0.0\\nN: 0\\n\",\"left\\nQ: 0.0\\nN: 0\\n\",\"right\\nQ: 0.0\\nN: 0\\n\",\"GridWorldState(4, 3, false)\\nN: 79\\n\",\"up\\nQ: -10.0\\nN: 20\\n\",\"down\\nQ: -10.0\\nN: 20\\n\",\"left\\nQ: -10.0\\nN: 20\\n\",\"right\\nQ: -10.0\\nN: 19\\n\",\"GridWorldState(6, 4, false)\\nN: 55\\n\",\"up\\nQ: -0.5964545454545456\\nN: 11\\n\",\"down\\nQ: 0.5719455264705883\\nN: 17\\n\",\"left\\nQ: -0.5607692307692308\\nN: 13\\n\",\"right\\nQ: -0.052071428571428435\\nN: 14\\n\",\"GridWorldState(3, 3, false)\\nN: 69\\n\",\"up\\nQ: -4.556775611250001\\nN: 8\\n\",\"down\\nQ: -1.3413945169565216\\nN: 23\\n\",\"left\\nQ: -0.3937991142857143\\nN: 35\\n\",\"right\\nQ: -9.0\\nN: 3\\n\",\"GridWorldState(10, 7, false)\\nN: 0\\n\",\"GridWorldState(9, 3, true)\\nN: 0\\n\",\"GridWorldState(9, 3, true)\\nN: 0\\n\",\"GridWorldState(9, 3, true)\\nN: 0\\n\",\"GridWorldState(9, 3, true)\\nN: 0\\n\",\"GridWorldState(2, 7, false)\\nN: 0\\n\",\"GridWorldState(4, 6, true)\\nN: 0\\n\",\"GridWorldState(4, 6, true)\\nN: 0\\n\",\"GridWorldState(4, 3, true)\\nN: 0\\n\",\"GridWorldState(4, 3, true)\\nN: 0\\n\",\"GridWorldState(4, 3, true)\\nN: 0\\n\",\"GridWorldState(4, 3, true)\\nN: 0\\n\"],\"link_style\":[\"\",\"\",\"stroke-width:5.2138724590670105px\",\"stroke-width:8.922536101660608px\",\"stroke-width:5.396870722208659px\",\"stroke-width:20.0px\",\"stroke-width:5.6188174428825155px\",\"stroke-width:5.97836755560688px\",\"stroke-width:5.402627528753511px\",\"stroke-width:20.0px\",\"\",\"stroke-width:6.679151072108546px\",\"stroke-width:4.97416003373807px\",\"stroke-width:4.499297039752018px\",\"stroke-width:20.0px\",\"\",\"stroke-width:NaNpx\",\"stroke-width:NaNpx\",\"stroke-width:NaNpx\",\"stroke-width:NaNpx\",\"\",\"stroke-width:NaNpx\",\"stroke-width:NaNpx\",\"stroke-width:NaNpx\",\"stroke-width:NaNpx\",\"\",\"stroke-width:14.770978917519928px\",\"stroke-width:8.528028654224418px\",\"stroke-width:20.0px\",\"stroke-width:10.44465935734187px\",\"\",\"stroke-width:14.484414758400552px\",\"stroke-width:10.736562541900378px\",\"stroke-width:6.2604379697819095px\",\"stroke-width:20.0px\",\"\",\"stroke-width:20.0px\",\"stroke-width:16.32993161855452px\",\"stroke-width:16.32993161855452px\",\"stroke-width:16.32993161855452px\",\"\",\"stroke-width:20.0px\",\"stroke-width:12.649110640673518px\",\"stroke-width:12.649110640673518px\",\"stroke-width:12.649110640673518px\",\"\",\"stroke-width:16.035674514745462px\",\"stroke-width:15.118578920369089px\",\"stroke-width:20.0px\",\"stroke-width:15.118578920369089px\",\"\",\"stroke-width:14.414999403128943px\",\"stroke-width:20.0px\",\"stroke-width:18.932552877030567px\",\"stroke-width:15.458364692328841px\",\"\",\"stroke-width:20.0px\",\"stroke-width:4.70716840598808px\",\"stroke-width:4.452535556994169px\",\"stroke-width:6.296836171649597px\",\"\",\"stroke-width:8.213654170497799px\",\"stroke-width:4.033353763771789px\",\"stroke-width:3.788676152120412px\",\"stroke-width:20.0px\",\"\",\"stroke-width:10.610965676722952px\",\"stroke-width:11.987647963787714px\",\"stroke-width:9.66091783079296px\",\"stroke-width:20.0px\",\"\",\"stroke-width:19.480430680228288px\",\"stroke-width:20.0px\",\"stroke-width:20.0px\",\"stroke-width:20.0px\",\"\",\"stroke-width:20.0px\",\"stroke-width:19.06925178491185px\",\"stroke-width:17.056057308448835px\",\"stroke-width:12.060453783110546px\",\"\",\"stroke-width:18.257418583505537px\",\"stroke-width:18.257418583505537px\",\"stroke-width:20.0px\",\"stroke-width:17.32050807568877px\",\"\",\"stroke-width:20.0px\",\"stroke-width:3.985083851093743px\",\"stroke-width:3.8585408471315463px\",\"stroke-width:11.745861422045882px\",\"\",\"stroke-width:13.228756555322953px\",\"stroke-width:15.0px\",\"stroke-width:14.142135623730951px\",\"stroke-width:20.0px\",\"\",\"stroke-width:16.842105263157894px\",\"stroke-width:6.046908048987398px\",\"stroke-width:5.048243708750231px\",\"stroke-width:20.0px\",\"\",\"stroke-width:20.0px\",\"stroke-width:18.74873733122184px\",\"stroke-width:19.69463855669324px\",\"stroke-width:20.0px\",\"\",\"stroke-width:12.853691738343029px\",\"stroke-width:20.0px\",\"stroke-width:11.03354568734741px\",\"stroke-width:16.151457061744964px\",\"\",\"stroke-width:15.811388300841898px\",\"stroke-width:20.0px\",\"stroke-width:18.708286933869708px\",\"stroke-width:19.525624189766635px\",\"\",\"stroke-width:20.0px\",\"stroke-width:20.0px\",\"stroke-width:20.0px\",\"stroke-width:20.0px\",\"\",\"stroke-width:2.7394690053949686px\",\"stroke-width:3.1234752377721215px\",\"stroke-width:2.873179332146128px\",\"stroke-width:20.0px\",\"\",\"stroke-width:16.32993161855452px\",\"stroke-width:19.148542155126762px\",\"stroke-width:18.257418583505537px\",\"stroke-width:20.0px\",\"\",\"stroke-width:13.09307341415954px\",\"stroke-width:10.690449676496975px\",\"stroke-width:18.516401995451027px\",\"stroke-width:20.0px\",\"\",\"stroke-width:20.0px\",\"stroke-width:0.0px\",\"stroke-width:0.0px\",\"stroke-width:0.0px\",\"\",\"stroke-width:20.0px\",\"stroke-width:0.0px\",\"stroke-width:0.0px\",\"stroke-width:0.0px\",\"\",\"stroke-width:11.547005383792515px\",\"stroke-width:20.0px\",\"stroke-width:16.32993161855452px\",\"stroke-width:16.32993161855452px\",\"\",\"stroke-width:9.085135251589959px\",\"stroke-width:10.079052613579393px\",\"stroke-width:20.0px\",\"stroke-width:7.126966450997983px\",\"\",\"stroke-width:5.190665196422049px\",\"stroke-width:8.874484586945758px\",\"stroke-width:6.597220328454717px\",\"stroke-width:20.0px\",\"\",\"stroke-width:17.580981459830653px\",\"stroke-width:17.056057308448835px\",\"stroke-width:20.0px\",\"stroke-width:18.586407545691703px\",\"\",\"stroke-width:13.743685418725535px\",\"stroke-width:17.950549357115012px\",\"stroke-width:15.634719199411434px\",\"stroke-width:20.0px\",\"\",\"stroke-width:20.0px\",\"stroke-width:20.0px\",\"stroke-width:0.0px\",\"stroke-width:0.0px\",\"\",\"stroke-width:20.0px\",\"stroke-width:2.4970005988000232px\",\"stroke-width:3.0971480654125543px\",\"stroke-width:2.726545939058017px\",\"\",\"stroke-width:NaNpx\",\"stroke-width:NaNpx\",\"stroke-width:NaNpx\",\"stroke-width:NaNpx\",\"\",\"stroke-width:20.0px\",\"stroke-width:20.0px\",\"stroke-width:0.0px\",\"stroke-width:0.0px\",\"\",\"stroke-width:NaNpx\",\"stroke-width:NaNpx\",\"stroke-width:NaNpx\",\"stroke-width:NaNpx\",\"\",\"stroke-width:6.31685652829212px\",\"stroke-width:4.627222948390527px\",\"stroke-width:4.185480638490935px\",\"stroke-width:20.0px\",\"\",\"stroke-width:15.19398377178095px\",\"stroke-width:7.250615737399726px\",\"stroke-width:20.0px\",\"stroke-width:6.233549779791837px\",\"\",\"stroke-width:17.019258867935264px\",\"stroke-width:17.811271131234424px\",\"stroke-width:20.0px\",\"stroke-width:13.896166675593024px\",\"\",\"stroke-width:20.0px\",\"stroke-width:0.0px\",\"stroke-width:0.0px\",\"stroke-width:0.0px\",\"\",\"stroke-width:12.649110640673518px\",\"stroke-width:20.0px\",\"stroke-width:12.649110640673518px\",\"stroke-width:12.649110640673518px\",\"\",\"stroke-width:14.889467836879975px\",\"stroke-width:5.32938710021193px\",\"stroke-width:4.070386632407063px\",\"stroke-width:20.0px\",\"\",\"stroke-width:20.0px\",\"stroke-width:11.547005383792515px\",\"stroke-width:9.428090415820634px\",\"stroke-width:11.547005383792515px\",\"\",\"stroke-width:20.0px\",\"stroke-width:8.48528137423857px\",\"stroke-width:9.797958971132712px\",\"stroke-width:8.48528137423857px\",\"\",\"stroke-width:11.547005383792515px\",\"stroke-width:20.0px\",\"stroke-width:11.547005383792515px\",\"stroke-width:11.547005383792515px\",\"\",\"stroke-width:3.3886949683494305px\",\"stroke-width:15.217718205053643px\",\"stroke-width:12.37376449779492px\",\"stroke-width:20.0px\",\"\",\"stroke-width:14.142135623730951px\",\"stroke-width:20.0px\",\"stroke-width:14.142135623730951px\",\"stroke-width:0.0px\",\"\",\"stroke-width:20.0px\",\"stroke-width:19.975414864708043px\",\"stroke-width:19.975414864708043px\",\"stroke-width:19.975414864708043px\",\"\",\"stroke-width:11.616357391030313px\",\"stroke-width:20.0px\",\"stroke-width:5.377328579378649px\",\"stroke-width:8.214005083884006px\",\"\",\"stroke-width:12.389735068004908px\",\"stroke-width:4.859653913846296px\",\"stroke-width:3.327177587852354px\",\"stroke-width:20.0px\",\"\",\"stroke-width:12.649110640673518px\",\"stroke-width:14.605934866804429px\",\"stroke-width:13.662601021279464px\",\"stroke-width:20.0px\",\"\",\"stroke-width:11.547005383792515px\",\"stroke-width:20.0px\",\"stroke-width:13.333333333333332px\",\"stroke-width:11.547005383792515px\",\"\",\"stroke-width:20.0px\",\"stroke-width:14.142135623730951px\",\"stroke-width:14.142135623730951px\",\"stroke-width:0.0px\",\"\",\"stroke-width:14.142135623730951px\",\"stroke-width:14.142135623730951px\",\"stroke-width:20.0px\",\"stroke-width:0.0px\",\"\",\"stroke-width:20.0px\",\"stroke-width:17.32050807568877px\",\"stroke-width:20.0px\",\"stroke-width:20.0px\",\"\",\"stroke-width:20.0px\",\"stroke-width:20.0px\",\"stroke-width:20.0px\",\"stroke-width:20.0px\",\"\",\"stroke-width:20.0px\",\"stroke-width:20.0px\",\"stroke-width:0.0px\",\"stroke-width:0.0px\",\"\",\"stroke-width:8.94427190999916px\",\"stroke-width:20.0px\",\"stroke-width:8.94427190999916px\",\"stroke-width:0.0px\",\"\",\"stroke-width:15.118578920369089px\",\"stroke-width:20.0px\",\"stroke-width:20.0px\",\"stroke-width:16.903085094570333px\",\"\",\"stroke-width:11.547005383792515px\",\"stroke-width:20.0px\",\"stroke-width:0.0px\",\"stroke-width:0.0px\",\"\",\"stroke-width:18.212833856189725px\",\"stroke-width:19.754591932991794px\",\"stroke-width:19.754591932991794px\",\"stroke-width:20.0px\",\"\",\"stroke-width:20.0px\",\"stroke-width:20.0px\",\"stroke-width:0.0px\",\"stroke-width:0.0px\",\"\",\"stroke-width:NaNpx\",\"stroke-width:NaNpx\",\"stroke-width:NaNpx\",\"stroke-width:NaNpx\",\"\",\"stroke-width:20.0px\",\"stroke-width:20.0px\",\"stroke-width:20.0px\",\"stroke-width:19.493588689617926px\",\"\",\"stroke-width:16.087993330796873px\",\"stroke-width:20.0px\",\"stroke-width:17.48949264390412px\",\"stroke-width:18.149704259460602px\",\"\",\"stroke-width:9.561828874675149px\",\"stroke-width:16.21286966755555px\",\"stroke-width:20.0px\",\"stroke-width:5.855400437691199px\",\"\",\"\",\"\",\"\",\"\",\"\",\"\",\"\",\"\",\"\",\"\",\"\"],\"title\":\"MCTS tree\",\"text\":[\"GridWorldState(4, 1, false)\\nN:   1667\\n\",\"GridWorldState(5, 2, false)\\nN:    552\\n\",\"                       up\\nQ:  -1.28\\nN:     28\\n\",\"                     down\\nQ:   0.91\\nN:     82\\n\",\"                     left\\nQ:  -1.38\\nN:     30\\n\",\"                    right\\nQ:   2.42\\nN:    412\\n\",\"                       up\\nQ:  -0.05\\nN:    106\\n\",\"                     down\\nQ:   0.00\\nN:    120\\n\",\"                     left\\nQ:  -0.24\\nN:     98\\n\",\"                    right\\nQ:   1.98\\nN:   1343\\n\",\"GridWorldState(5, 1, false)\\nN:   1306\\n\",\"                       up\\nQ:   1.25\\nN:    119\\n\",\"                     down\\nQ:   0.00\\nN:     66\\n\",\"                     left\\nQ:   0.01\\nN:     54\\n\",\"                    right\\nQ:   2.90\\nN:   1067\\n\",\"GridWorldState(9, 8, false)\\nN:      0\\n\",\"                       up\\nQ:   0.00\\nN:      0\\n\",\"                     down\\nQ:   0.00\\nN:      0\\n\",\"                     left\\nQ:   0.00\\nN:      0\\n\",\"                    right\\nQ:   0.00\\nN:      0\\n\",\"GridWorldState(7, 9, false)\\nN:      0\\n\",\"                       up\\nQ:   0.00\\nN:      0\\n\",\"                     down\\nQ:   0.00\\nN:      0\\n\",\"                     left\\nQ:   0.00\\nN:      0\\n\",\"                    right\\nQ:   0.00\\nN:      0\\n\",\"GridWorldState(1, 5, false)\\nN:     22\\n\",\"                       up\\nQ:   0.00\\nN:      6\\n\",\"                     down\\nQ:   0.00\\nN:      2\\n\",\"                     left\\nQ:   0.00\\nN:     11\\n\",\"                    right\\nQ:   0.00\\nN:      3\\n\",\"GridWorldState(6, 2, false)\\nN:    663\\n\",\"                       up\\nQ:   3.12\\nN:    182\\n\",\"                     down\\nQ:   2.35\\nN:    100\\n\",\"                     left\\nQ:   0.32\\nN:     34\\n\",\"                    right\\nQ:   3.59\\nN:    347\\n\",\"GridWorldState(9, 6, false)\\nN:      9\\n\",\"                       up\\nQ:   1.59\\nN:      3\\n\",\"                     down\\nQ:   0.00\\nN:      2\\n\",\"                     left\\nQ:   0.00\\nN:      2\\n\",\"                    right\\nQ:   0.00\\nN:      2\\n\",\"GridWorldState(2, 5, false)\\nN:     11\\n\",\"                       up\\nQ:   0.00\\nN:      5\\n\",\"                     down\\nQ:   0.00\\nN:      2\\n\",\"                     left\\nQ:   0.00\\nN:      2\\n\",\"                    right\\nQ:   0.00\\nN:      2\\n\",\"GridWorldState(1, 4, false)\\nN:     39\\n\",\"                       up\\nQ:  -0.59\\nN:      9\\n\",\"                     down\\nQ:   0.00\\nN:      8\\n\",\"                     left\\nQ:   0.00\\nN:     14\\n\",\"                    right\\nQ:   0.00\\nN:      8\\n\",\"GridWorldState(3, 2, false)\\nN:    232\\n\",\"                       up\\nQ:  -1.11\\nN:     40\\n\",\"                     down\\nQ:   0.00\\nN:     77\\n\",\"                     left\\nQ:  -0.13\\nN:     69\\n\",\"                    right\\nQ:  -0.81\\nN:     46\\n\",\"GridWorldState(9, 1, false)\\nN:    413\\n\",\"                       up\\nQ:   5.81\\nN:    343\\n\",\"                     down\\nQ:   0.00\\nN:     19\\n\",\"                     left\\nQ:   1.09\\nN:     17\\n\",\"                    right\\nQ:   2.63\\nN:     34\\n\",\"GridWorldState(6, 1, false)\\nN:   1041\\n\",\"                       up\\nQ:   2.45\\nN:    141\\n\",\"                     down\\nQ:   0.00\\nN:     34\\n\",\"                     left\\nQ:  -0.61\\nN:     30\\n\",\"                    right\\nQ:   3.67\\nN:    836\\n\",\"GridWorldState(3, 1, false)\\nN:    506\\n\",\"                       up\\nQ:  -0.25\\nN:     76\\n\",\"                     down\\nQ:   0.00\\nN:     97\\n\",\"                     left\\nQ:  -0.55\\nN:     63\\n\",\"                    right\\nQ:   1.20\\nN:    270\\n\",\"GridWorldState(1, 1, false)\\nN:    154\\n\",\"                       up\\nQ:   0.00\\nN:     37\\n\",\"                     down\\nQ:   0.00\\nN:     39\\n\",\"                     left\\nQ:   0.00\\nN:     39\\n\",\"                    right\\nQ:   0.00\\nN:     39\\n\",\"GridWorldState(2, 4, false)\\nN:     33\\n\",\"                       up\\nQ:   0.00\\nN:     11\\n\",\"                     down\\nQ:   0.00\\nN:     10\\n\",\"                     left\\nQ:   0.00\\nN:      8\\n\",\"                    right\\nQ:  -3.30\\nN:      4\\n\",\"GridWorldState(5, 4, false)\\nN:     41\\n\",\"                       up\\nQ:  -0.24\\nN:     10\\n\",\"                     down\\nQ:  -0.42\\nN:     10\\n\",\"                     left\\nQ:  -0.30\\nN:     12\\n\",\"                    right\\nQ:  -0.59\\nN:      9\\n\",\"GridWorldState(8, 1, false)\\nN:    573\\n\",\"                       up\\nQ:   5.31\\nN:    403\\n\",\"                     down\\nQ:   0.00\\nN:     16\\n\",\"                     left\\nQ:  -0.14\\nN:     15\\n\",\"                    right\\nQ:   4.52\\nN:    139\\n\",\"GridWorldState(7, 5, false)\\nN:     40\\n\",\"                       up\\nQ:  -0.25\\nN:      7\\n\",\"                     down\\nQ:   0.95\\nN:      9\\n\",\"                     left\\nQ:   0.00\\nN:      8\\n\",\"                    right\\nQ:   0.85\\nN:     16\\n\",\"GridWorldState(7, 2, false)\\nN:    673\\n\",\"                       up\\nQ:   4.60\\nN:    256\\n\",\"                     down\\nQ:   1.58\\nN:     33\\n\",\"                     left\\nQ:   0.66\\nN:     23\\n\",\"                    right\\nQ:   4.73\\nN:    361\\n\",\"GridWorldState(1, 2, false)\\nN:    127\\n\",\"                       up\\nQ:   0.06\\nN:     33\\n\",\"                     down\\nQ:  -0.15\\nN:     29\\n\",\"                     left\\nQ:   0.00\\nN:     32\\n\",\"                    right\\nQ:   0.08\\nN:     33\\n\",\"GridWorldState(7, 4, false)\\nN:    109\\n\",\"                       up\\nQ:   1.38\\nN:     19\\n\",\"                     down\\nQ:   3.52\\nN:     46\\n\",\"                     left\\nQ:   0.52\\nN:     14\\n\",\"                    right\\nQ:   2.62\\nN:     30\\n\",\"GridWorldState(2, 1, false)\\nN:    221\\n\",\"                       up\\nQ:  -0.39\\nN:     40\\n\",\"                     down\\nQ:   0.00\\nN:     64\\n\",\"                     left\\nQ:   0.00\\nN:     56\\n\",\"                    right\\nQ:   0.32\\nN:     61\\n\",\"GridWorldState(7, 6, false)\\nN:      8\\n\",\"                       up\\nQ:   0.00\\nN:      2\\n\",\"                     down\\nQ:   0.00\\nN:      2\\n\",\"                     left\\nQ:   0.00\\nN:      2\\n\",\"                    right\\nQ:   0.00\\nN:      2\\n\",\"GridWorldState(8, 3, false)\\nN:    567\\n\",\"                       up\\nQ:   0.00\\nN:     10\\n\",\"                     down\\nQ:   1.60\\nN:     13\\n\",\"                     left\\nQ:  -0.26\\nN:     11\\n\",\"                    right\\nQ:   7.63\\nN:    533\\n\",\"GridWorldState(8, 5, false)\\nN:     41\\n\",\"                       up\\nQ:   0.00\\nN:      8\\n\",\"                     down\\nQ:   0.71\\nN:     11\\n\",\"                     left\\nQ:   0.00\\nN:     10\\n\",\"                    right\\nQ:   0.81\\nN:     12\\n\",\"GridWorldState(4, 4, false)\\nN:     18\\n\",\"                       up\\nQ:  -4.35\\nN:      3\\n\",\"                     down\\nQ:  -9.00\\nN:      2\\n\",\"                     left\\nQ:  -0.80\\nN:      6\\n\",\"                    right\\nQ:   0.00\\nN:      7\\n\",\"GridWorldState(5, 7, false)\\nN:      1\\n\",\"                       up\\nQ:   0.00\\nN:      1\\n\",\"                     down\\nQ:   0.00\\nN:      0\\n\",\"                     left\\nQ:   0.00\\nN:      0\\n\",\"                    right\\nQ:   0.00\\nN:      0\\n\",\"GridWorldState(7, 8, false)\\nN:      1\\n\",\"                       up\\nQ:   0.00\\nN:      1\\n\",\"                     down\\nQ:   0.00\\nN:      0\\n\",\"                     left\\nQ:   0.00\\nN:      0\\n\",\"                    right\\nQ:   0.00\\nN:      0\\n\",\"GridWorldState(10, 5, false)\\nN:      8\\n\",\"                       up\\nQ:   0.00\\nN:      1\\n\",\"                     down\\nQ:   2.43\\nN:      3\\n\",\"                     left\\nQ:   0.00\\nN:      2\\n\",\"                    right\\nQ:   0.00\\nN:      2\\n\",\"GridWorldState(10, 3, false)\\nN:    100\\n\",\"                       up\\nQ:   3.79\\nN:     13\\n\",\"                     down\\nQ:   4.95\\nN:     16\\n\",\"                     left\\nQ:   7.98\\nN:     63\\n\",\"                    right\\nQ:   0.00\\nN:      8\\n\",\"GridWorldState(6, 3, false)\\nN:    265\\n\",\"                       up\\nQ:  -1.13\\nN:     13\\n\",\"                     down\\nQ:   1.37\\nN:     38\\n\",\"                     left\\nQ:   0.08\\nN:     21\\n\",\"                    right\\nQ:   3.78\\nN:    193\\n\",\"GridWorldState(2, 3, false)\\nN:     74\\n\",\"                       up\\nQ:  -0.52\\nN:     17\\n\",\"                     down\\nQ:  -0.74\\nN:     16\\n\",\"                     left\\nQ:   0.00\\nN:     22\\n\",\"                    right\\nQ:  -0.35\\nN:     19\\n\",\"GridWorldState(8, 4, false)\\nN:    104\\n\",\"                       up\\nQ:   1.51\\nN:     17\\n\",\"                     down\\nQ:   3.71\\nN:     29\\n\",\"                     left\\nQ:   2.23\\nN:     22\\n\",\"                    right\\nQ:   3.51\\nN:     36\\n\",\"GridWorldState(10, 6, false)\\nN:      2\\n\",\"                       up\\nQ:   0.00\\nN:      1\\n\",\"                     down\\nQ:   0.00\\nN:      1\\n\",\"                     left\\nQ:   0.00\\nN:      0\\n\",\"                    right\\nQ:   0.00\\nN:      0\\n\",\"GridWorldState(9, 2, false)\\nN:   1765\\n\",\"                       up\\nQ:   7.96\\nN:   1668\\n\",\"                     down\\nQ:   3.23\\nN:     26\\n\",\"                     left\\nQ:   4.23\\nN:     40\\n\",\"                    right\\nQ:   3.03\\nN:     31\\n\",\"GridWorldState(3, 6, false)\\nN:      0\\n\",\"                       up\\nQ:   0.00\\nN:      0\\n\",\"                     down\\nQ:   0.00\\nN:      0\\n\",\"                     left\\nQ:   0.00\\nN:      0\\n\",\"                    right\\nQ:   0.00\\nN:      0\\n\",\"GridWorldState(8, 7, false)\\nN:      2\\n\",\"                       up\\nQ:   0.00\\nN:      1\\n\",\"                     down\\nQ:   0.00\\nN:      1\\n\",\"                     left\\nQ:   0.00\\nN:      0\\n\",\"                    right\\nQ:   0.00\\nN:      0\\n\",\"GridWorldState(1, 7, false)\\nN:      0\\n\",\"                       up\\nQ:   0.00\\nN:      0\\n\",\"                     down\\nQ:   0.00\\nN:      0\\n\",\"                     left\\nQ:   0.00\\nN:      0\\n\",\"                    right\\nQ:   0.00\\nN:      0\\n\",\"GridWorldState(7, 3, false)\\nN:    492\\n\",\"                       up\\nQ:   2.54\\nN:     41\\n\",\"                     down\\nQ:   1.19\\nN:     22\\n\",\"                     left\\nQ:  -1.39\\nN:     18\\n\",\"                    right\\nQ:   5.53\\nN:    411\\n\",\"GridWorldState(10, 2, false)\\nN:    316\\n\",\"                       up\\nQ:   5.77\\nN:    101\\n\",\"                     down\\nQ:   2.80\\nN:     23\\n\",\"                     left\\nQ:   6.41\\nN:    175\\n\",\"                    right\\nQ:   0.00\\nN:     17\\n\",\"GridWorldState(1, 3, false)\\nN:     87\\n\",\"                       up\\nQ:   0.00\\nN:     21\\n\",\"                     down\\nQ:   0.00\\nN:     23\\n\",\"                     left\\nQ:   0.00\\nN:     29\\n\",\"                    right\\nQ:  -1.19\\nN:     14\\n\",\"GridWorldState(9, 7, false)\\nN:      1\\n\",\"                       up\\nQ:   0.00\\nN:      1\\n\",\"                     down\\nQ:   0.00\\nN:      0\\n\",\"                     left\\nQ:   0.00\\nN:      0\\n\",\"                    right\\nQ:   0.00\\nN:      0\\n\",\"GridWorldState(9, 5, false)\\nN:     22\\n\",\"                       up\\nQ:   0.00\\nN:      4\\n\",\"                     down\\nQ:   2.00\\nN:     10\\n\",\"                     left\\nQ:   0.00\\nN:      4\\n\",\"                    right\\nQ:   0.00\\nN:      4\\n\",\"GridWorldState(8, 2, false)\\nN:    845\\n\",\"                       up\\nQ:   5.87\\nN:    281\\n\",\"                     down\\nQ:   2.36\\nN:     36\\n\",\"                     left\\nQ:   1.47\\nN:     21\\n\",\"                    right\\nQ:   6.28\\nN:    507\\n\",\"GridWorldState(5, 5, false)\\nN:     17\\n\",\"                       up\\nQ:   0.00\\nN:      9\\n\",\"                     down\\nQ:   0.00\\nN:      3\\n\",\"                     left\\nQ:  -4.98\\nN:      2\\n\",\"                    right\\nQ:  -0.89\\nN:      3\\n\",\"GridWorldState(10, 1, false)\\nN:    160\\n\",\"                       up\\nQ:   4.53\\nN:    100\\n\",\"                     down\\nQ:   0.00\\nN:     18\\n\",\"                     left\\nQ:   2.78\\nN:     24\\n\",\"                    right\\nQ:   0.00\\nN:     18\\n\",\"GridWorldState(4, 5, false)\\nN:      6\\n\",\"                       up\\nQ:  -4.50\\nN:      1\\n\",\"                     down\\nQ:   0.00\\nN:      3\\n\",\"                     left\\nQ:  -6.56\\nN:      1\\n\",\"                    right\\nQ:  -4.50\\nN:      1\\n\",\"GridWorldState(4, 2, false)\\nN:    416\\n\",\"                       up\\nQ:  -8.48\\nN:      6\\n\",\"                     down\\nQ:  -0.37\\nN:    121\\n\",\"                     left\\nQ:  -0.92\\nN:     80\\n\",\"                    right\\nQ:   0.30\\nN:    209\\n\",\"GridWorldState(3, 5, false)\\nN:      4\\n\",\"                       up\\nQ:  -5.90\\nN:      1\\n\",\"                     down\\nQ:   0.00\\nN:      2\\n\",\"                     left\\nQ:  -7.29\\nN:      1\\n\",\"                    right\\nQ:   0.00\\nN:      0\\n\",\"GridWorldState(9, 3, false)\\nN:   1625\\n\",\"                       up\\nQ:  10.00\\nN:    407\\n\",\"                     down\\nQ:  10.00\\nN:    406\\n\",\"                     left\\nQ:  10.00\\nN:    406\\n\",\"                    right\\nQ:  10.00\\nN:    406\\n\",\"GridWorldState(5, 3, false)\\nN:    131\\n\",\"                       up\\nQ:  -1.51\\nN:     28\\n\",\"                     down\\nQ:   0.58\\nN:     83\\n\",\"                     left\\nQ:  -4.62\\nN:      6\\n\",\"                    right\\nQ:  -3.15\\nN:     14\\n\",\"GridWorldState(7, 1, false)\\nN:    797\\n\",\"                       up\\nQ:   3.96\\nN:    208\\n\",\"                     down\\nQ:   0.00\\nN:     32\\n\",\"                     left\\nQ:  -1.34\\nN:     15\\n\",\"                    right\\nQ:   4.50\\nN:    542\\n\",\"GridWorldState(10, 4, false)\\nN:     36\\n\",\"                       up\\nQ:   0.00\\nN:      6\\n\",\"                     down\\nQ:   4.05\\nN:      8\\n\",\"                     left\\nQ:   2.86\\nN:      7\\n\",\"                    right\\nQ:   0.00\\nN:     15\\n\",\"GridWorldState(9, 4, false)\\nN:     38\\n\",\"                       up\\nQ:   2.30\\nN:      6\\n\",\"                     down\\nQ:   6.50\\nN:     18\\n\",\"                     left\\nQ:   4.08\\nN:      8\\n\",\"                    right\\nQ:   2.20\\nN:      6\\n\",\"GridWorldState(6, 6, false)\\nN:      4\\n\",\"                       up\\nQ:   0.00\\nN:      2\\n\",\"                     down\\nQ:   0.00\\nN:      1\\n\",\"                     left\\nQ:  -4.05\\nN:      1\\n\",\"                    right\\nQ:   0.00\\nN:      0\\n\",\"GridWorldState(2, 6, false)\\nN:      4\\n\",\"                       up\\nQ:   0.00\\nN:      1\\n\",\"                     down\\nQ:   0.00\\nN:      1\\n\",\"                     left\\nQ:   0.00\\nN:      2\\n\",\"                    right\\nQ:   0.00\\nN:      0\\n\",\"GridWorldState(8, 6, false)\\nN:     15\\n\",\"                       up\\nQ:   0.61\\nN:      4\\n\",\"                     down\\nQ:   0.00\\nN:      3\\n\",\"                     left\\nQ:   0.00\\nN:      4\\n\",\"                    right\\nQ:   1.08\\nN:      4\\n\",\"GridWorldState(6, 5, false)\\nN:     16\\n\",\"                       up\\nQ:  -0.91\\nN:      4\\n\",\"                     down\\nQ:   0.00\\nN:      4\\n\",\"                     left\\nQ:  -0.91\\nN:      4\\n\",\"                    right\\nQ:   0.00\\nN:      4\\n\",\"GridWorldState(4, 6, false)\\nN:      2\\n\",\"                       up\\nQ:  -5.00\\nN:      1\\n\",\"                     down\\nQ:  -5.00\\nN:      1\\n\",\"                     left\\nQ:   0.00\\nN:      0\\n\",\"                    right\\nQ:   0.00\\nN:      0\\n\",\"GridWorldState(5, 6, false)\\nN:      7\\n\",\"                       up\\nQ:  -2.95\\nN:      1\\n\",\"                     down\\nQ:   0.00\\nN:      5\\n\",\"                     left\\nQ:  -4.50\\nN:      1\\n\",\"                    right\\nQ:   0.00\\nN:      0\\n\",\"GridWorldState(3, 4, false)\\nN:     23\\n\",\"                       up\\nQ:  -3.67\\nN:      4\\n\",\"                     down\\nQ:  -1.37\\nN:      7\\n\",\"                     left\\nQ:  -1.16\\nN:      7\\n\",\"                    right\\nQ:  -3.24\\nN:      5\\n\",\"GridWorldState(1, 6, false)\\nN:      4\\n\",\"                       up\\nQ:   0.00\\nN:      1\\n\",\"                     down\\nQ:   0.00\\nN:      3\\n\",\"                     left\\nQ:   0.00\\nN:      0\\n\",\"                    right\\nQ:   0.00\\nN:      0\\n\",\"GridWorldState(2, 2, false)\\nN:    155\\n\",\"                       up\\nQ:  -0.40\\nN:     34\\n\",\"                     down\\nQ:  -0.05\\nN:     40\\n\",\"                     left\\nQ:   0.00\\nN:     40\\n\",\"                    right\\nQ:  -0.08\\nN:     41\\n\",\"GridWorldState(7, 7, false)\\nN:      2\\n\",\"                       up\\nQ:   0.00\\nN:      1\\n\",\"                     down\\nQ:   0.00\\nN:      1\\n\",\"                     left\\nQ:   0.00\\nN:      0\\n\",\"                    right\\nQ:   0.00\\nN:      0\\n\",\"GridWorldState(6, 7, false)\\nN:      0\\n\",\"                       up\\nQ:   0.00\\nN:      0\\n\",\"                     down\\nQ:   0.00\\nN:      0\\n\",\"                     left\\nQ:   0.00\\nN:      0\\n\",\"                    right\\nQ:   0.00\\nN:      0\\n\",\"GridWorldState(4, 3, false)\\nN:     79\\n\",\"                       up\\nQ: -10.00\\nN:     20\\n\",\"                     down\\nQ: -10.00\\nN:     20\\n\",\"                     left\\nQ: -10.00\\nN:     20\\n\",\"                    right\\nQ: -10.00\\nN:     19\\n\",\"GridWorldState(6, 4, false)\\nN:     55\\n\",\"                       up\\nQ:  -0.60\\nN:     11\\n\",\"                     down\\nQ:   0.57\\nN:     17\\n\",\"                     left\\nQ:  -0.56\\nN:     13\\n\",\"                    right\\nQ:  -0.05\\nN:     14\\n\",\"GridWorldState(3, 3, false)\\nN:     69\\n\",\"                       up\\nQ:  -4.56\\nN:      8\\n\",\"                     down\\nQ:  -1.34\\nN:     23\\n\",\"                     left\\nQ:  -0.39\\nN:     35\\n\",\"                    right\\nQ:  -9.00\\nN:      3\\n\",\"GridWorldState(10, 7, false)\\nN:      0\\n\",\"GridWorldState(9, 3, true)\\nN:      0\\n\",\"GridWorldState(9, 3, true)\\nN:      0\\n\",\"GridWorldState(9, 3, true)\\nN:      0\\n\",\"GridWorldState(9, 3, true)\\nN:      0\\n\",\"GridWorldState(2, 7, false)\\nN:      0\\n\",\"GridWorldState(4, 6, true)\\nN:      0\\n\",\"GridWorldState(4, 6, true)\\nN:      0\\n\",\"GridWorldState(4, 3, true)\\nN:      0\\n\",\"GridWorldState(4, 3, true)\\nN:      0\\n\",\"GridWorldState(4, 3, true)\\nN:      0\\n\",\"GridWorldState(4, 3, true)\\nN:      0\\n\"],\"style\":[\"\",\"\",\"stroke:#903800\",\"stroke:#744600\",\"stroke:#913700\",\"stroke:#614F00\",\"stroke:#804000\",\"stroke:#7F4000\",\"stroke:#833E00\",\"stroke:#664D00\",\"\",\"stroke:#704800\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#5B5200\",\"\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"\",\"stroke:#585400\",\"stroke:#624F00\",\"stroke:#7B4200\",\"stroke:#525700\",\"\",\"stroke:#6B4A00\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"\",\"stroke:#873C00\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"\",\"stroke:#8E3900\",\"stroke:#7F4000\",\"stroke:#813F00\",\"stroke:#8A3B00\",\"\",\"stroke:#356500\",\"stroke:#7F4000\",\"stroke:#724700\",\"stroke:#5E5100\",\"\",\"stroke:#605000\",\"stroke:#7F4000\",\"stroke:#873C00\",\"stroke:#515700\",\"\",\"stroke:#833E00\",\"stroke:#7F4000\",\"stroke:#863D00\",\"stroke:#704800\",\"\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#AA2B00\",\"\",\"stroke:#833E00\",\"stroke:#853D00\",\"stroke:#833E00\",\"stroke:#873C00\",\"\",\"stroke:#3C6200\",\"stroke:#7F4000\",\"stroke:#813F00\",\"stroke:#465D00\",\"\",\"stroke:#833E00\",\"stroke:#734600\",\"stroke:#7F4000\",\"stroke:#754500\",\"\",\"stroke:#455D00\",\"stroke:#6B4A00\",\"stroke:#774400\",\"stroke:#435E00\",\"\",\"stroke:#7F4000\",\"stroke:#813F00\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"\",\"stroke:#6E4900\",\"stroke:#535600\",\"stroke:#794300\",\"stroke:#5E5100\",\"\",\"stroke:#843E00\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7B4200\",\"\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"\",\"stroke:#7F4000\",\"stroke:#6B4A00\",\"stroke:#833E00\",\"stroke:#1E7100\",\"\",\"stroke:#7F4000\",\"stroke:#764500\",\"stroke:#7F4000\",\"stroke:#754500\",\"\",\"stroke:#B72400\",\"stroke:#F20700\",\"stroke:#8A3B00\",\"stroke:#7F4000\",\"\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"\",\"stroke:#7F4000\",\"stroke:#614F00\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"\",\"stroke:#4F5800\",\"stroke:#406000\",\"stroke:#1A7300\",\"stroke:#7F4000\",\"\",\"stroke:#8E3900\",\"stroke:#6E4900\",\"stroke:#7F4000\",\"stroke:#4F5800\",\"\",\"stroke:#863D00\",\"stroke:#893B00\",\"stroke:#7F4000\",\"stroke:#843E00\",\"\",\"stroke:#6C4A00\",\"stroke:#505800\",\"stroke:#634E00\",\"stroke:#535600\",\"\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"\",\"stroke:#1A7300\",\"stroke:#565500\",\"stroke:#4A5B00\",\"stroke:#595300\",\"\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"\",\"stroke:#5F5000\",\"stroke:#704800\",\"stroke:#913700\",\"stroke:#396300\",\"\",\"stroke:#366500\",\"stroke:#5C5200\",\"stroke:#2E6900\",\"stroke:#7F4000\",\"\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#8F3800\",\"\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"\",\"stroke:#7F4000\",\"stroke:#664D00\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"\",\"stroke:#356500\",\"stroke:#614F00\",\"stroke:#6D4900\",\"stroke:#2F6800\",\"\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#BF2000\",\"stroke:#8B3A00\",\"\",\"stroke:#465D00\",\"stroke:#7F4000\",\"stroke:#5C5200\",\"stroke:#7F4000\",\"\",\"stroke:#B92300\",\"stroke:#7F4000\",\"stroke:#D31600\",\"stroke:#B92300\",\"\",\"stroke:#EC0A00\",\"stroke:#843E00\",\"stroke:#8B3A00\",\"stroke:#7C4200\",\"\",\"stroke:#CB1A00\",\"stroke:#7F4000\",\"stroke:#DC1200\",\"stroke:#7F4000\",\"\",\"stroke:#008000\",\"stroke:#008000\",\"stroke:#008000\",\"stroke:#008000\",\"\",\"stroke:#933600\",\"stroke:#784400\",\"stroke:#BA2300\",\"stroke:#A82C00\",\"\",\"stroke:#4D5900\",\"stroke:#7F4000\",\"stroke:#913700\",\"stroke:#465D00\",\"\",\"stroke:#7F4000\",\"stroke:#4C5A00\",\"stroke:#5B5200\",\"stroke:#7F4000\",\"\",\"stroke:#624F00\",\"stroke:#2D6900\",\"stroke:#4B5A00\",\"stroke:#634E00\",\"\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#B32600\",\"stroke:#7F4000\",\"\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"\",\"stroke:#784400\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#724700\",\"\",\"stroke:#8B3A00\",\"stroke:#7F4000\",\"stroke:#8B3A00\",\"stroke:#7F4000\",\"\",\"stroke:#BF2000\",\"stroke:#BF2000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"\",\"stroke:#A52D00\",\"stroke:#7F4000\",\"stroke:#B92300\",\"stroke:#7F4000\",\"\",\"stroke:#AE2900\",\"stroke:#913700\",\"stroke:#8E3900\",\"stroke:#A92B00\",\"\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"\",\"stroke:#853D00\",\"stroke:#804000\",\"stroke:#7F4000\",\"stroke:#804000\",\"\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"stroke:#7F4000\",\"\",\"stroke:#FF0000\",\"stroke:#FF0000\",\"stroke:#FF0000\",\"stroke:#FF0000\",\"\",\"stroke:#873C00\",\"stroke:#784400\",\"stroke:#873C00\",\"stroke:#804000\",\"\",\"stroke:#BA2300\",\"stroke:#913700\",\"stroke:#853D00\",\"stroke:#F20700\",\"\",\"\",\"\",\"\",\"\",\"\",\"\",\"\",\"\",\"\",\"\",\"\"]}};\n",
       "        var rootID = 1-1;\n",
       "        var div = \"treeviseNszNgLP\";\n",
       "        var initExpand = 0\n",
       "        var initDuration = 750\n",
       "        var svgHeight = 600\n",
       "        if (typeof $ === 'undefined') {\n",
       "    loadScript(\"https://code.jquery.com/jquery-3.1.1.min.js\", run);\n",
       "} else {\n",
       "    run();\n",
       "}\n",
       "\n",
       "function run() {\n",
       "    if (typeof d3 === 'undefined') {\n",
       "        loadScript(\"https://d3js.org/d3.v3.js\", showTree);\n",
       "    } else {\n",
       "        showTree();\n",
       "    }\n",
       "}\n",
       "\n",
       "function loadScript(url, callback)\n",
       "{\n",
       "    console.log(\"starting script load...\")\n",
       "    // Adding the script tag to the head as suggested before\n",
       "    var head = document.getElementsByTagName('head')[0];\n",
       "    var script = document.createElement('script');\n",
       "    script.type = 'text/javascript';\n",
       "    script.src = url;\n",
       "\n",
       "    // Then bind the event to the callback function.\n",
       "    // There are several events for cross browser compatibility.\n",
       "    script.onreadystatechange = callback;\n",
       "    script.onload = callback;\n",
       "\n",
       "    // Fire the loading\n",
       "    head.appendChild(script);\n",
       "}\n",
       "\n",
       "function showTree() {\n",
       "        \n",
       "    // var margin = {top: 20, right: 120, bottom: 20, left: 120},\n",
       "    var margin = {top: 20, right: 120, bottom: 80, left: 120},\n",
       "        width = $(\"#\"+div).width() - margin.right - margin.left,\n",
       "        height = svgHeight - margin.top - margin.bottom;\n",
       "        // TODO make height a parameter of TreeVisualizer\n",
       "\n",
       "    var i = 0,\n",
       "        root;\n",
       "\n",
       "    var tree = d3.layout.tree()\n",
       "        .size([width, height]);\n",
       "\n",
       "    var diagonal = d3.svg.diagonal();\n",
       "        //.projection(function(d) { return [d.y, d.x]; });\n",
       "        // uncomment above to make the tree go horizontally\n",
       "\n",
       "    // see http://stackoverflow.com/questions/16265123/resize-svg-when-window-is-resized-in-d3-js\n",
       "    if (d3.select(\"#\"+div+\"_svg\").empty()) {\n",
       "        $(\".d3twarn\").remove();\n",
       "        d3.select(\"#\"+div).append(\"svg\")\n",
       "            .attr(\"id\", div+\"_svg\")\n",
       "            .attr(\"width\", width + margin.right + margin.left)\n",
       "            .attr(\"height\", height + margin.top + margin.bottom);\n",
       "    }\n",
       "\n",
       "    d3.select(\"#\"+div+\"_svg\").selectAll(\"*\").remove();\n",
       "\n",
       "    var svg = d3.select(\"#\"+div+\"_svg\")\n",
       "        .append(\"g\")\n",
       "            .attr(\"transform\", \"translate(\" + margin.left + \",\" + margin.top + \")\");\n",
       "\n",
       "    // console.log(\"tree data:\");\n",
       "    // console.log(treeData[rootID]);\n",
       "    root = createDisplayNode(rootID, initExpand);\n",
       "    root.x0 = width / 2;\n",
       "    root.y0 = 0;\n",
       "\n",
       "    update(root, initDuration);\n",
       "    console.log(\"tree should appear\");\n",
       "\n",
       "    function createDisplayNode(id, expandLevel) {\n",
       "      var dnode = {\"dataID\":id,\n",
       "                   \"children\": null,\n",
       "                   \"_children\":null};\n",
       "      if (expandLevel > 0) {\n",
       "        initializeChildren(dnode, expandLevel);\n",
       "      }\n",
       "      return dnode;\n",
       "    }\n",
       "\n",
       "    function initializeChildren(d, expandLevel) {\n",
       "      // create children\n",
       "      var children = treeData.children[d.dataID];\n",
       "      d.children = [];\n",
       "      if (children) {\n",
       "        for (var i = 0; i < children.length; i++) {\n",
       "          var cid = children[i]-1;\n",
       "          d.children.push(createDisplayNode(cid, expandLevel-1));\n",
       "        }\n",
       "      }\n",
       "    }\n",
       "\n",
       "    /**\n",
       "     * Recursively called to update each node in the tree.\n",
       "     * \n",
       "     * source is a d3 node that has position, etc.\n",
       "    */\n",
       "    function update(source, duration) {\n",
       "\n",
       "      width = $(\"#\"+div).width() - margin.right - margin.left,\n",
       "      height = $(\"#\"+div).height() - margin.top - margin.bottom;\n",
       "\n",
       "      tree.size([width,height]);\n",
       "      d3.select(\"#\"+div).attr(\"width\", width + margin.right + margin.left)\n",
       "            .attr(\"height\", height + margin.top + margin.bottom);\n",
       "      d3.select(\"#\"+div+\"_svg\").attr(\"width\", width + margin.right + margin.left)\n",
       "             .attr(\"height\", height + margin.top + margin.bottom);\n",
       "\n",
       "\n",
       "      // Compute the new tree layout.\n",
       "      var nodes = tree.nodes(root).reverse(),\n",
       "          links = tree.links(nodes);\n",
       "\n",
       "\n",
       "      // Update the nodes…\n",
       "      var node = svg.selectAll(\"g.node\")\n",
       "          .data(nodes, function(d) { return d.id || (d.id = ++i); });\n",
       "\n",
       "      // Enter any new nodes at the parent's previous position.\n",
       "      var nodeEnter = node.enter().append(\"g\")\n",
       "          .attr(\"class\", \"node\")\n",
       "          .attr(\"transform\", function(d) { return \"translate(\" + source.x0 + \",\" + source.y0 + \")\"; })\n",
       "          .on(\"click\", click)\n",
       "\n",
       "      nodeEnter.append(\"circle\")\n",
       "          .attr(\"r\", \"10px\")\n",
       "          .attr(\"style\", function(d) { return treeData.style[d.dataID]; } )\n",
       "\n",
       "      var tbox = nodeEnter.append(\"text\")\n",
       "          .attr(\"y\", 25)\n",
       "          .attr(\"text-anchor\", \"middle\")\n",
       "          //.text( function(d) { return treeData.text[d.dataID]; } )\n",
       "          .style(\"fill-opacity\", 1e-6);\n",
       "\n",
       "      tbox.each( function(d) {\n",
       "          var el = d3.select(this)\n",
       "          var text = treeData.text[d.dataID];\n",
       "          var lines = text.split('\\n');\n",
       "          for (var i = 0; i < lines.length; i++) {\n",
       "              var tspan = el.append(\"tspan\").text(lines[i]);\n",
       "              if (i > 0) {\n",
       "                  tspan.attr(\"x\", 0).attr(\"dy\", \"1.2em\");\n",
       "              }\n",
       "          }\n",
       "      });\n",
       "\n",
       "      //tooltip\n",
       "      nodeEnter.append(\"title\").text( function(d) { return treeData.tooltip[d.dataID];} );\n",
       "\n",
       "      // Transition nodes to their new position.\n",
       "      var nodeUpdate = node.transition()\n",
       "          .duration(duration)\n",
       "          .attr(\"transform\", function(d) { return \"translate(\" + d.x + \",\" + d.y + \")\"; });\n",
       "\n",
       "      nodeUpdate.select(\"text\")\n",
       "          .style(\"fill-opacity\", 1);\n",
       "\n",
       "      // Transition exiting nodes to the parent's new position.\n",
       "      var nodeExit = node.exit().transition()\n",
       "          .duration(duration)\n",
       "          .attr(\"transform\", function(d) { return \"translate(\" + source.x + \",\" + source.y + \")\"; })\n",
       "          .remove();\n",
       "\n",
       "      nodeExit.select(\"text\")\n",
       "          .style(\"fill-opacity\", 1e-6);\n",
       "\n",
       "      // Update the links…\n",
       "      var link = svg.selectAll(\"path.link\")\n",
       "          .data(links, function(d) { return d.target.id; });\n",
       "\n",
       "      // Enter any new links at the parent's previous position.\n",
       "      // XXX link width should be based on transition data, not node data\n",
       "      link.enter().insert(\"path\", \"g\")\n",
       "          .attr(\"class\", \"link\")\n",
       "          .attr(\"style\", function(d) {\n",
       "              var ls = treeData.link_style[d.target.dataID];\n",
       "              return ls;\n",
       "          } )\n",
       "          .attr(\"d\", function(d) {\n",
       "            var o = {x: source.x0, y: source.y0};\n",
       "            return diagonal({source: o, target: o});\n",
       "          });\n",
       "\n",
       "      // Transition links to their new position.\n",
       "      link.transition()\n",
       "          .duration(duration)\n",
       "          .attr(\"d\", diagonal);\n",
       "\n",
       "      // Transition exiting nodes to the parent's new position.\n",
       "      link.exit().transition()\n",
       "          .duration(duration)\n",
       "          .attr(\"d\", function(d) {\n",
       "            var o = {x: source.x, y: source.y};\n",
       "            return diagonal({source: o, target: o});\n",
       "          })\n",
       "          .remove();\n",
       "\n",
       "      // Stash the old positions for transition.\n",
       "      nodes.forEach(function(d) {\n",
       "        d.x0 = d.x;\n",
       "        d.y0 = d.y;\n",
       "      });\n",
       "    }\n",
       "\n",
       "    // Toggle children on click.\n",
       "    function click(d) {\n",
       "      if (d.children) {\n",
       "        d._children = d.children;\n",
       "        d.children = null;\n",
       "      } else if (d._children) {\n",
       "        d.children = d._children;\n",
       "        d._children = null;\n",
       "      } else {\n",
       "        initializeChildren(d, 1);\n",
       "      }\n",
       "      update(d, 750);\n",
       "    }\n",
       "\n",
       "}\n",
       "\n",
       "        })();\n",
       "    </script>\n",
       "    <p class=\"d3twarn\">\n",
       "    Attempting to display the tree. If the tree is large, this may take some time.\n",
       "    </p>\n",
       "    <p class=\"d3twarn\">\n",
       "    Note: D3Trees.jl requires an internet connection. If no tree appears, please check your connection. To help fix this, please see <a href=\"https://github.com/sisl/D3Trees.jl/issues/10\">this issue</a>. You may also diagnose errors with the javascript console (Ctrl-Shift-J in chrome).\n",
       "    </p>\n",
       "    </div>\n",
       "    </body>\n",
       "    </html>\n",
       "\n"
      ],
      "text/plain": [
       "GridWorldState(4, 1, false)\n",
       "N:   1667\n",
       "\n",
       "├──                       up\n",
       "│  Q:  -0.05\n",
       "│  N:    106\n",
       "\n",
       "│  ├──GridWorldState(4, 2, false)\n",
       "│  │  N:    416\n",
       "\n",
       "│  │  ├──                       up\n",
       "│  │  │  Q:  -8.48\n",
       "│  │  │  N:      6\n",
       " (2 children)\n",
       "│  │  ├──                     down\n",
       "│  │  │  Q:  -0.37\n",
       "│  │  │  N:    121\n",
       " (4 children)\n",
       "│  │  ├──                     left\n",
       "│  │  │  Q:  -0.92\n",
       "│  │  │  N:     80\n",
       " (4 children)\n",
       "│  │  └──                    right\n",
       "│  │     Q:   0.30\n",
       "│  │     N:    209\n",
       " (4 children)\n",
       "│  ├──GridWorldState(3, 1, false)\n",
       "│  │  N:    506\n",
       "\n",
       "│  │  ├──                       up\n",
       "│  │  │  Q:  -0.25\n",
       "│  │  │  N:     76\n",
       " (3 children)\n",
       "│  │  ├──                     down\n",
       "│  │  │  Q:   0.00\n",
       "│  │  │  N:     97\n",
       " (1 children)\n",
       "│  │  ├──                     left\n",
       "│  │  │  Q:  -0.55\n",
       "│  │  │  N:     63\n",
       " (3 children)\n",
       "│  │  └──                    right\n",
       "│  │     Q:   1.20\n",
       "│  │     N:    270\n",
       " (3 children)\n",
       "│  └──GridWorldState(5, 1, false)\n",
       "│     N:   1306\n",
       "\n",
       "│     ├──                       up\n",
       "│     │  Q:   1.25\n",
       "│     │  N:    119\n",
       " (3 children)\n",
       "│     ├──                     down\n",
       "│     │  Q:   0.00\n",
       "│     │  N:     66\n",
       " (1 children)\n",
       "│     ├──                     left\n",
       "│     │  Q:   0.01\n",
       "│     │  N:     54\n",
       " (3 children)\n",
       "│     └──                    right\n",
       "│        Q:   2.90\n",
       "│        N:   1067\n",
       " (3 children)\n",
       "├──                     down\n",
       "│  Q:   0.00\n",
       "│  N:    120\n",
       "\n",
       "│  └──GridWorldState(4, 1, false)\n",
       "│     N:   1667\n",
       "\n",
       "│     ├──                       up\n",
       "│     │  Q:  -0.05\n",
       "│     │  N:    106\n",
       " (3 children)\n",
       "│     ├──                     down\n",
       "│     │  Q:   0.00\n",
       "│     │  N:    120\n",
       " (1 children)\n",
       "│     ├──                     left\n",
       "│     │  Q:  -0.24\n",
       "│     │  N:     98\n",
       " (3 children)\n",
       "│     └──                    right\n",
       "│        Q:   1.98\n",
       "│        N:   1343\n",
       " (3 children)\n",
       "├──                     left\n",
       "│  Q:  -0.24\n",
       "│  N:     98\n",
       "\n",
       "│  ├──GridWorldState(4, 2, false)\n",
       "│  │  N:    416\n",
       "\n",
       "│  │  ├──                       up\n",
       "│  │  │  Q:  -8.48\n",
       "│  │  │  N:      6\n",
       " (2 children)\n",
       "│  │  ├──                     down\n",
       "│  │  │  Q:  -0.37\n",
       "│  │  │  N:    121\n",
       " (4 children)\n",
       "│  │  ├──                     left\n",
       "│  │  │  Q:  -0.92\n",
       "│  │  │  N:     80\n",
       " (4 children)\n",
       "│  │  └──                    right\n",
       "│  │     Q:   0.30\n",
       "│  │     N:    209\n",
       " (4 children)\n",
       "│  ├──GridWorldState(3, 1, false)\n",
       "│  │  N:    506\n",
       "\n",
       "│  │  ├──                       up\n",
       "│  │  │  Q:  -0.25\n",
       "│  │  │  N:     76\n",
       " (3 children)\n",
       "│  │  ├──                     down\n",
       "│  │  │  Q:   0.00\n",
       "│  │  │  N:     97\n",
       " (1 children)\n",
       "│  │  ├──                     left\n",
       "│  │  │  Q:  -0.55\n",
       "│  │  │  N:     63\n",
       " (3 children)\n",
       "│  │  └──                    right\n",
       "│  │     Q:   1.20\n",
       "│  │     N:    270\n",
       " (3 children)\n",
       "│  └──GridWorldState(5, 1, false)\n",
       "│     N:   1306\n",
       "\n",
       "│     ├──                       up\n",
       "│     │  Q:   1.25\n",
       "│     │  N:    119\n",
       " (3 children)\n",
       "│     ├──                     down\n",
       "│     │  Q:   0.00\n",
       "│     │  N:     66\n",
       " (1 children)\n",
       "│     ├──                     left\n",
       "│     │  Q:   0.01\n",
       "│     │  N:     54\n",
       " (3 children)\n",
       "│     └──                    right\n",
       "│        Q:   2.90\n",
       "│        N:   1067\n",
       " (3 children)\n",
       "└──                    right\n",
       "   Q:   1.98\n",
       "   N:   1343\n",
       "\n",
       "   ├──GridWorldState(4, 2, false)\n",
       "   │  N:    416\n",
       "\n",
       "   │  ├──                       up\n",
       "   │  │  Q:  -8.48\n",
       "   │  │  N:      6\n",
       " (2 children)\n",
       "   │  ├──                     down\n",
       "   │  │  Q:  -0.37\n",
       "   │  │  N:    121\n",
       " (4 children)\n",
       "   │  ├──                     left\n",
       "   │  │  Q:  -0.92\n",
       "   │  │  N:     80\n",
       " (4 children)\n",
       "   │  └──                    right\n",
       "   │     Q:   0.30\n",
       "   │     N:    209\n",
       " (4 children)\n",
       "   ├──GridWorldState(3, 1, false)\n",
       "   │  N:    506\n",
       "\n",
       "   │  ├──                       up\n",
       "   │  │  Q:  -0.25\n",
       "   │  │  N:     76\n",
       " (3 children)\n",
       "   │  ├──                     down\n",
       "   │  │  Q:   0.00\n",
       "   │  │  N:     97\n",
       " (1 children)\n",
       "   │  ├──                     left\n",
       "   │  │  Q:  -0.55\n",
       "   │  │  N:     63\n",
       " (3 children)\n",
       "   │  └──                    right\n",
       "   │     Q:   1.20\n",
       "   │     N:    270\n",
       " (3 children)\n",
       "   └──GridWorldState(5, 1, false)\n",
       "      N:   1306\n",
       "\n",
       "      ├──                       up\n",
       "      │  Q:   1.25\n",
       "      │  N:    119\n",
       " (3 children)\n",
       "      ├──                     down\n",
       "      │  Q:   0.00\n",
       "      │  N:     66\n",
       " (1 children)\n",
       "      ├──                     left\n",
       "      │  Q:   0.01\n",
       "      │  N:     54\n",
       " (3 children)\n",
       "      └──                    right\n",
       "         Q:   2.90\n",
       "         N:   1067\n",
       " (3 children)\n"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "using D3Trees\n",
    "\n",
    "# first, run the planner on the state\n",
    "s = state_hist(hist)[1]\n",
    "a = action(planner, s)\n",
    "\n",
    "# show the tree (click the node to expand)\n",
    "D3Tree(planner, s)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Julia 0.6.1",
   "language": "julia",
   "name": "julia-0.6"
  },
  "language_info": {
   "file_extension": ".jl",
   "mimetype": "application/julia",
   "name": "julia",
   "version": "0.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
